Resiliency director

ABSTRACT

Systems and methods of orchestrating recoveries of virtual machines protected by a data management systems from a primary system to a secondary system, such that performing the recoveries depends on relationships between the virtual machines. First data indicative of a recovery plan associated with a failover of at least one group of virtual machines is received. The recovery plan includes an application group with data indicative of a hierarchical relationship between the virtual machines wherein each of the virtual machines is associated with an order based on the second data. A plurality of sequences is created in the application group to designate an order of executing a plurality of recoveries for each of the virtual machines. A first recovery is executed in parallel for each of the virtual machines associated with a first sequence and a subsequent recovery is executed in parallel for each of a subsequent set of sequences.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. ProvisionalApplication Ser. No. 62/013,334 entitled RESILIENCY DIRECTOR, filed Jun.17, 2014, and which is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

This invention relates generally to data management, data protection,disaster recovery and business continuity. More specifically, thisinvention relates to a system and method for providing techniques toorchestrate failover and test failover.

BACKGROUND

The business requirements for managing the lifecycle of application datahave been traditionally met by deploying multiple point solutions, eachof which addresses a part of the lifecycle. This has resulted in acomplex and expensive infrastructure where multiple copies of data arecreated and moved multiple times to individual storage repositories. Theadoption of server virtualization has become a catalyst for simple,agile and low-cost compute infrastructure. This has led to largerdeployments of virtual hosts and storage, further exacerbating the gapbetween the emerging compute models and the current data managementimplementations.

Applications that provide business services depend on storage of theirdata at various stages of its lifecycle. FIG. 1 shows a typical set ofdata management operations that would be applied to the data of anapplication such as a database underlying a business service such aspayroll management. In order to provide a business service, application102 requires primary data storage 122 with some contracted level ofreliability and availability.

Backups 104 are made to guard against corruption or the primary datastorage through hardware or software failure or human error. Typicallybackups may be made daily or weekly to local disk or tape 124, and movedless frequently (weekly or monthly) to a remote physically securelocation 125.

Concurrent development and test 106 of new applications based on thesame database requires a development team to have access to another copyof the data 126. Such a snapshot might be made weekly, depending ondevelopment schedules.

Compliance with legal or voluntary policies 108 may require that somedata be retained for safely future access for some number of years;usually data is copied regularly (say, monthly) to a long-term archivingsystem 128.

Disaster Recovery services 110 guard against catastrophic loss of dataif systems providing primary business services fail due to some physicaldisaster. Primary data is copied 130 to a physically distinct locationas frequently as is feasible given other constraints (such as cost). Inthe event of a disaster the primary site can be reconstructed and datamoved back from the safe copy.

Business Continuity services 112 provide a facility for ensuringcontinued business services should the primary site become compromised.Usually this requires a hot copy 132 of the primary data that is innear-lockstep with the primary data, as well as duplicate systems andapplications and mechanisms for switching incoming requests to theBusiness Continuity servers.

Thus, data management is currently a collection of point applicationsmanaging the different parts of the lifecycle. This has been an artifactof evolution of data management solutions over the last two decades.

Current Data Management architecture and implementations such asdescribed above involve multiple applications addressing different partsof data lifecycle management, all of them performing certain commonfunctions: (a) make a copy of application data (the frequency of thisaction is commonly termed the Recovery Point Objective (RPO)), (b) storethe copy of data in an exclusive storage repository, typically in aproprietary format, and (c) retain the copy for certain duration,measured as Retention Time. A primary difference in each of the pointsolutions is in the frequency of the RPO, the Retention Time, and thecharacteristics of the individual storage repositories used, includingcapacity, cost and geographic location.

In a series of prior patent applications, e.g., U.S. Ser. No.12/947,375, a system and method for managing data has been presentedthat uses Data Management Virtualization. Data Management activities,such as Backup, Replication and Archiving are virtualized in that theydo not have to be configured and run individually and separately.Instead, the user defines their business requirement with regard to thelifecycle of the data, and the Data Management Virtualization Systemperforms these operations automatically. A snapshot is taken fromprimary storage to secondary storage; this snapshot is then used for abackup operation to other secondary storage. Essentially an arbitrarynumber of these backups may be made, providing a level of dataprotection specified by a Service Level Agreement.

The present application provides enhancements to the above system fordata management virtualization.

SUMMARY

Systems and methods are disclosed for orchestrating recoveries ofvirtual machines protected by a plurality of data management systemsfrom a primary system to a secondary system, such that performing therecoveries depends on relationships between the virtual machines. Insome embodiments, the systems and methods disclosed herein includereceiving first data indicative of a recovery plan associated with afailover of at least one group of virtual machines, each of the at leastone groups of virtual machines protected by a data management systemassociated with the at least one group of virtual machines. In someembodiments, the recovery plan includes an application group, theapplication group including second data indicative of a hierarchicalrelationship between the virtual machines in the at least one group ofvirtual machines, wherein each of the virtual machines in the at leastone group of virtual machines is associated with an order based on thesecond data. In some embodiments, the systems and methods disclosedherein include creating a plurality of sequences in the applicationgroup to designate an order of executing a plurality of recoveries foreach of the virtual machines in the at least one group of virtualmachines, wherein the plurality of sequences are ordered for executionbased on a relationship between the virtual machines contained with theplurality of sequences. In some embodiments, the systems and methodsdisclosed herein include executing in parallel a first recovery for eachof the virtual machines associated with a first sequence. In someembodiments, the systems and methods disclosed herein include executingin parallel, by the computing device, a subsequent recovery for each ofa subsequent set of sequences in order when the application groupincludes a subsequent sequence, such that an order of performing therecoveries depends on the relationships between the virtual machines.

In some embodiments, the systems and methods disclosed herein includeupdating the recovery plan with results of the failover, the results ofthe failover comprising a successful recovery result, a partiallysuccessful recovery result, or a failed recovery result. In someembodiments, the systems and methods disclosed herein includedetermining a status of the failover for each of the virtual machines;terminating an execution of the plurality of recoveries when the statusis associated with a critical failure; and updating the recovery planwith the failed recovery result.

In some embodiments, the recovery plan further includes at least one ofa hypervisor associated with the failover, a collector associated withthe failover; a customer associated with the failover, a data managementsystem associated with the failover, a resource pool associated with thefailover; and a port group associated with the failover. In someembodiments, the failover is a test failover.

In some embodiments, the systems and methods disclosed herein includeresetting the recovery plan, wherein resetting the recovery plancomprises receiving third data indicative of a recovery plan resetrequest; executing a delete operation for virtual machines associatedwith a last sequence when an execution of the recovery plan associatedwith a reset is a test failover; and executing, by the computing device,a delete operation for virtual machines associated with each of a priorsequence when an execution of the recovery plan associated with a resetis a test failover when the application group includes a prior sequence.

In some embodiments, the systems and methods disclosed herein includeupdating the recovery plan with a ready to run status. In someembodiments each of the at least one data management systems protect theplurality of virtual machines using dedup async replication, the dedupasync replication providing a remote copy of fourth data associated withthe virtual machines, in which remote replication is provided withreduced bandwidth requirements by copying deduplicated differences indata from a local storage site to a disaster recovery site. In someembodiments, the systems and methods disclosed herein are used invirtual machine by virtual machine recoveries independent of productionsource architecture, while using replication storage-basedmethodologies.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of current methods deployed to manage thedata lifecycle for a business service.

FIG. 2 is an overview of the management of data throughout its lifecycleby a single Data Management Virtualization System.

FIG. 3 is a simplified block diagram of the Data ManagementVirtualization system.

FIG. 4 is a view of the Data Management Virtualization Engine.

FIG. 5 illustrates the Object Management and Data Movement Engine.

FIG. 6 shows the Storage Pool Manager.

FIG. 7 shows the decomposition of the Service Level Agreement.

FIG. 8 illustrates the Application Specific Module.

FIG. 9 shows the Service Policy Manager.

FIG. 10 is a flowchart of the Service Policy Scheduler.

FIG. 11 is a block diagram of the Content Addressable Storage (CAS)provider.

FIG. 12 shows the definition of an object handle within the CAS system.

FIG. 13 shows the data model and operations for the temporalrelationship graph stored for objects within the CAS.

FIG. 14 is a diagram representing the operation of a garbage collectionalgorithm in the CAS.

FIG. 15 is a flowchart for the operation of copying an object into theCAS.

FIG. 16 is a system diagram of a typical deployment of the DataManagement Virtualization system.

FIG. 17 is a schematic diagram of a characteristic physical serverdevice for use with the Data Management Virtualization system.

FIG. 18 is a schematic diagram showing the data model for a datafingerprint to be used in conjunction with certain embodiments of theinvention.

FIG. 19 is a system architecture diagram of a deployment of the DataManagement Virtualization system that incorporates data fingerprinting.

FIG. 20 is a process diagram for the operation of copying an objectusing a hybrid seeding algorithm.

FIG. 21 is a process diagram for the operation of a Data ManagementVirtualization system that provides replication for business continuity.

FIG. 22 is a system diagram showing the components involved in an RDdeployment, according to some embodiments of the present disclosure.

FIG. 23 is a component diagram depicting an RD Collector, according tosome embodiments of the present disclosure.

FIG. 24 is a component diagram depicting an RD Server, according to someembodiments of the present disclosure.

FIG. 25 is a component diagram depicting a recovery plan 2403, accordingto some embodiments of the present disclosure.

FIG. 26 is a flow diagram illustrating a process of setting up RD toperform failovers and test failovers, according to some embodiments ofthe present disclosure.

FIG. 27 is a flow diagram illustrating a process of executing a recoveryplan of either failover or test failover execution type, according tosome embodiments of the present disclosure.

FIG. 28 is a flow diagram illustrating a process of resetting a recoveryplan for either failover or test failover execution type, according tosome embodiments of the present disclosure.

DETAILED DESCRIPTION

This disclosure pertains to generating a data fingerprint for an objectstored in a virtual storage pool that may be used to compare two objectsover the life of those data objects.

This disclosure also pertains to a method for improved incremental copyperformance using hybrid seeding to perform copies and differencingoperations using different virtual storage pools.

This disclosure also pertains to a mechanism for data replication fordisaster recovery and business continuity using a pipeline of storagepools.

In the Data Management Virtualization system described below, a userdefines business requirements with regard to the lifecycle of the data,and the Data Management Virtualization System performs these operationsautomatically. A snapshot is taken from primary storage to secondarystorage; this snapshot is then used for a backup operation to othersecondary storage. Essentially an arbitrary number of these backups maybe made, providing a level of data protection specified by a ServiceLevel Agreement.

The data management engine is operable to execute a sequence of snapshotoperations to create point-in-time images of application data on a firststorage pool, each successive point-in-time image corresponding to aspecific, successive time-state of the application data, and eachsnapshot operation creating difference information indicating whichapplication data has changed and the content of the changed applicationdata for the corresponding time state. The data management engine isalso operable to execute at least one back-up function for theapplication data that is scheduled for execution at non-consecutivetime-states, and is also full of maintain history information havingtime-state information indicating the time-state of the last back-upfunction performed on the application data for a corresponding back-upcopy of data. The data management engine creates composite differenceinformation from the difference information for each time-state betweenthe time-state of the last back-up function performed on the applicationdata and the time-state of the currently-scheduled back-up function tobe performed on the application data, and sends the composite differenceinformation to a second storage pool to be compiled with the back-upcopy of data at the last time-state to create a back-up copy of data forthe current time-state.

Data Management Virtualization technology according to this disclosureis based on an architecture and implementation based on the followingguiding principles.

First, define the business requirements of an application with a ServiceLevel Agreement (SLA) for its entire data lifecycle. The SLA is morethan a single RPO, Retention and Recovery Time Objective (RTO). Itdescribes the data protection characteristics for each stage of the datalifecycle. Each application may have a different SLA.

Second, provide a unified Data Management Virtualization Engine thatmanages the data protection lifecycle, moving data across the variousstorage repositories, with improved storage capacity and networkbandwidth. The Data Management Virtualization system achieves theseimprovements by leveraging extended capabilities of modern storagesystems by tracking the portions of the data that have changed over timeand by data deduplication and compression algorithms that reduce theamount of data that needs to be copied and moved.

Third, leverage a single master copy of the application data to be thebasis for multiple elements within the lifecycle. Many of the DataManagement operations such as backup, archival and replication depend ona stable, consistent copy of the data to be protected. The DataManagement Virtualization System leverages a single copy of the data formultiple purposes. A single instance of the data maintained by thesystem may serve as the source, from which each data management functionmay make additional copies as needed. This contrasts with requiringapplication data to be copied multiple times by multiple independentdata management applications in the traditional approach.

Fourth, abstracting physical storage resources into a series of dataprotection storage pools, which are virtualized out of different classesof storage including local and remote disk, solid state memory, tape andoptical media, private, public and/or hybrid storage clouds. The storagepools provide access independent of the type, physical location orunderlying storage technology. Business requirements for the lifecycleof data may call for copying the data to different types of storagemedia at different times. The Data Management Virtualization systemallows the user to classify and aggregate different storage media intostorage pools, for example, a Quick Recovery Pool, which may includehigh speed disks, and a Cost Efficient Long-term Storage Pool, which maybe a deduplicated store on high capacity disks, or a tape library. TheData Management Virtualization System can move data amongst these poolsto take advantage of the unique characteristics of each storage medium.The abstraction of Storage Pools provides access independent of thetype, physical location or underlying storage technology.

Fifth, improve the movement of the data between storage pools anddisaster locations utilizing underlying device capabilities andpost-deduplicated application data. The Data Management VirtualizationSystem discovers the capabilities of the storage systems that includethe Storage Pools, and takes advantage of these capabilities to movedata efficiently. If the Storage System is a disk array that supportsthe capability of creating a snapshot or clone of a data volume, theData Management Virtualization System will take advantage of thiscapability and use a snapshot to make a copy of the data rather thanreading the data from one place and writing it to another. Similarly, ifa storage system supports change tracking, the Data ManagementVirtualization System will update an older copy with just the changes toefficiently create a new copy. When moving data across a network, theData Management Virtualization system uses a deduplication andcompression algorithm that avoids sending data that is already availableon the other side of the network.

One key aspect of improving data movement is recognizing thatapplication data changes slowly over time. A copy of an application thatis made today will, in general, have a lot of similarities to the copyof the same application that was made yesterday. In fact today's copy ofthe data could be represented as yesterday's copy with a series of deltatransformations, where the size of the delta transformations themselvesare usually much smaller than all of the data in the copy itself. TheData Management Virtualization system captures and records thesetransformations in the form of bitmaps or extent lists. In oneembodiment of the system, the underlying storage resources—a disk arrayor server virtualization system—are capable of tracking the changes madeto a volume or file; in these environments, the Data ManagementVirtualization system queries the storage resources to obtain thesechange lists, and saves them with the data being protected.

In the preferred embodiment of the Data Management Virtualizationsystem, there is a mechanism for eavesdropping on the primary dataaccess path of the application, which enables the Data ManagementVirtualization system to observe which parts of the application data aremodified, and to generate its own bitmap of modified data. If, forexample, the application modifies blocks 100, 200 and 300 during aparticular period, the Data Management Virtualization system willeavesdrop on these events, and create a bitmap that indicates that theseparticular blocks were modified. When processing the next copy ofapplication data, the Data Management Virtualization system will onlyprocess blocks 100, 200 and 300 since it knows that these were the onlyblocks that were modified.

In one embodiment of the system, where the primary storage for theapplication is a modern disk array or storage virtualization appliance,the Data Management Virtualization system takes advantage of apoint-in-time snapshot capability of an underlying storage device tomake the initial copy of the data. This virtual copy mechanism is afast, efficient and low-impact technique of creating the initial copythat does not guarantee that all the bits will be copied, or storedtogether. Instead, virtual copies are constructed by maintainingmetadata and data structures, such as copy-on-write volume bitmaps orextents, that allow the copies to be reconstructed at access time. Thecopy has a lightweight impact on the application and on the primarystorage device. In another embodiment, where the application is based ona Server Virtualization System such as VMware or Xen, the DataManagement Virtualization system uses the similarvirtual-machine-snapshot capability that is built into the ServerVirtualization systems. When a virtual copy capability is not available,the Data Management Virtualization System may include its own built-insnapshot mechanism.

It is possible to use the snapshot as a data primitive underlying all ofthe data management functions supported by the system. Because it islightweight, the snapshot can be used as an internal operation even whenthe requested operation is not a snapshot per se; it is created toenable and facilitate other operations.

At the time of creation of a snapshot, there may be certain preparatoryoperations involved in order to create a coherent snapshot or coherentimage, such that the image may be restored to a state that is usable bythe application. These preparatory operations need only be performedonce, even if the snapshot will be leveraged across multiple datamanagement functions in the system, such as backup copies which arescheduled according to a policy. The preparatory operations may includeapplication quiescence, which includes flushing data caches and freezingthe state of the application; it may also include other operations knownin the art and other operations useful for retaining a complete image,such as collecting metadata information from the application to bestored with the image.

FIG. 2 illustrates one way that a Virtualized Data Management system canaddress the data lifecycle requirements described earlier in accordancewith these principles.

To serve local backup requirements, a sequence of efficient snapshotsare made within local high-availability storage 202. Some of thesesnapshots are used to serve development/test requirements without makinganother copy. For longer term retention of local backup, a copy is madeefficiently into long-term local storage 204, which in thisimplementation uses deduplication to reduce repeated copying. The copieswithin long-term storage may be accessed as backups or treated as anarchive, depending on the retention policy applied by the SLA. A copy ofthe data is made to remote storage 206 in order to satisfy requirementsfor remote backup and business continuity—again a single set of copiessuffices both purposes. As an alternative for remote backup and disasterrecovery, a further copy of the data may be made efficiently to arepository 208 hosted by a commercial or private cloud storage provider.

The Data Management Virtualization System

FIG. 3 illustrates the high level components of the Data ManagementVirtualization System that implements the above principles. Preferably,the system includes these basic functional components further describedbelow.

Application 300 creates and owns the data. This is the software systemthat has been deployed by the user, as for example, an email system, adatabase system, or financial reporting system, in order to satisfy somecomputational need. The Application typically runs on a server andutilizes storage. For illustrative purposes, only one application hasbeen indicated. In reality there may be hundreds or even thousands ofapplications that are managed by a single Data Management VirtualizationSystem.

Storage Resources 302 is where application data is stored through itslifecycle. The Storage Resources are the physical storage assets,including internal disk drives, disk arrays, optical and tape storagelibraries and cloud-based storage systems that the user has acquired toaddress data storage requirements. The storage resources include PrimaryStorage 310, where the online, active copy of the application data isstored, and Secondary Storage 312 where additional copies of theapplication data are stored for the purposes such as backup, disasterrecovery, archiving, indexing, reporting and other uses. Secondarystorage resources may include additional storage within the sameenclosure as the primary storage, as well as storage based on similar ordifferent storage technologies within the same data center, anotherlocation or across the internet.

One or more Management Workstations 308 allow the user to specify aService Level Agreement (SLA) 304 that defines the lifecycle for theapplication data. A Management workstation is a desktop or laptopcomputer or a mobile computing device that is used to configure, monitorand control the Data Management Virtualization System. A Service LevelAgreement is a detailed specification that captures the detailedbusiness requirements related to the creation, retention and deletion ofsecondary copies of the application data. The SLA is more than thesimple RTO and RPO that are used in traditional data managementapplications to represent the frequency of copies and the anticipatedrestore time for a single class of secondary storage. The SLA capturesthe multiple stages in the data lifecycle specification, and allows fornon-uniform frequency and retention specifications within each class ofsecondary storage. The SLA is described in greater detail in FIG. 7.

Data Management Virtualization Engine 306 manages all of the lifecycleof the application data as specified in SLA. It manages potentially alarge number of SLAs for a large number of applications. The DataManagement Virtualization Engine takes inputs from the user through theManagement Workstation and interacts with the applications to discoverthe applications primary storage resources. The Data ManagementVirtualization Engine makes decisions regarding what data needs to beprotected and what secondary storage resources best fulfill theprotection needs. For example, if an enterprise designates itsaccounting data as requiring copies to be made at very short intervalsfor business continuity purposes as well as for backup purposes, theEngine may decide to create copies of the accounting data at a shortinterval to a first storage pool, and to also create backup copies ofthe accounting data to a second storage pool at a longer interval,according to an appropriate set of SLAs. This is determined by thebusiness requirements of the storage application.

The Engine then makes copies of application data using advancedcapabilities of the storage resources as available. In the aboveexample, the Engine may schedule the short-interval business continuitycopy using a storage appliance's built-in virtual copy or snapshotcapabilities. The Data Management Virtualization Engine moves theapplication data amongst the storage resources in order to satisfy thebusiness requirements that are captured in the SLA. The Data ManagementVirtualization Engine is described in greater detail in FIG. 4.

The Data Management Virtualization System as a whole may be deployedwithin a single host computer system or appliance, or it may be onelogical entity but physically distributed across a network ofgeneral-purpose and purpose-built systems. Certain components of thesystem may also be deployed within a computing or storage cloud.

In one embodiment of the Data Management Virtualization System the DataManagement Virtualization Engine largely runs as multiple processes on afault tolerant, redundant pair of computers. Certain components of theData Management Virtualization Engine may run close to the applicationwithin the application servers. Some other components may run close tothe primary and secondary storage, within the storage fabric or in thestorage systems themselves. The Management stations are typicallydesktop and laptop computers and mobile devices that connect over asecure network to the Engine.

The Data Management Virtualization Engine

FIG. 4 illustrates an architectural overview of the Data ManagementVirtualization Engine 306 according to certain embodiments of theinvention. The 306 Engine includes the following modules:

Application Specific Module 402. This module is responsible forcontrolling and collecting metadata from the application 300.Application metadata includes information about the application such asthe type of application, details about its configuration, location ofits datastores, its current operating state. Controlling the operationof the application includes actions such as flushing cached data todisk, freezing and thawing application I/O, rotating or truncating logfiles, and shutting down and restarting applications. The ApplicationSpecific module performs these operations and sends and receivesmetadata in responses to commands from the Service Level Policy Engine406, described below. The Application Specific Module is described inmore detail in connection with FIG. 8.

Service Level Policy Engine 406. This module acts on the SLA 304provided by the user to make decisions regarding the creation, movementand deletion of copies of the application data. Each SLA describes thebusiness requirements related to protection of one application. TheService Level Policy Engine analyzes each SLA and arrives at a series ofactions each of which involve the copying of application data from onestorage location to another. The Service Level Policy Engine thenreviews these actions to determine priorities and dependencies, andschedules and initiates the data movement jobs. The Service Level PolicyEngine is described in more detail in connection with FIG. 9.

Object Manager and Data Movement Engine 410. This module creates acomposite object consisting of the Application data, the ApplicationMetadata and the SLA which it moves through different storage pools perinstruction from the Policy Engine. The Object Manager receivesinstructions from the Service Policy Engine 406 in the form of a commandto create a copy of application data in a particular pool based on thelive primary data 413 belonging to the application 300, or from anexisting copy, e.g., 415, in another pool. The copy of the compositeobject that is created by the Object Manager and the Data MovementEngine is self contained and self describing in that it contains notonly application data, but also application metadata and the SLA for theapplication. The Object Manager and Data Movement Engine are describedin more detail in connection with FIG. 5.

Storage Pool Manager 412. This module is a component that adapts andabstracts the underlying physical storage resources 302 and presentsthem as virtual storage pools 418. The physical storage resources arethe actual storage assets, such as disk arrays and tape libraries thatthe user has deployed for the purpose of supporting the lifecycle of thedata of the user's applications. These storage resources might be basedon different storage technologies such as disk, tape, flash memory oroptical storage. The storage resources may also have differentgeographic locations, cost and speed attributes, and may supportdifferent protocols. The role of the Storage Pool Manager is to combineand aggregate the storage resources, and mask the differences betweentheir programming interfaces. The Storage Pool Manager presents thephysical storage resources to the Object Manager 410 as a set of storagepools that have characteristics that make these pools suitable forparticular stages in the lifecycle of application data. The Storage PoolManager is described in more detail in connection with FIG. 6.

Object Manager and Data Movement Engine

FIG. 5 illustrates the Object Manager and Data Movement Engine 410. TheObject Manager and Data Movement Engine discovers and uses VirtualStorage Resources 510 presented to it by the Pool Managers 504. Itaccepts requests from the Service Level Policy Engine 406 to create andmaintain Data Storage Object instances from the resources in a VirtualStorage Pool, and it copies application data among instances of storageobjects from the Virtual Storage Pools according to the instructionsfrom the Service Level Policy Engine. The target pool selected for thecopy implicitly designates the business operation being selected, e.g.backup, replication or restore. The Service Level Policy Engine resideseither locally to the Object Manager (on the same system) or remotely,and communicates using a protocol over standard networkingcommunication. TCP/IP may be used in a preferred embodiment, as it iswell understood, widely available, and allows the Service Level PolicyEngine to be located locally to the Object Manager or remotely withlittle modification.

In one embodiment, the system may deploy the Service Level Policy Engineon the same computer system as the Object Manager for ease ofimplementation. In another embodiment, the system may employ multiplesystems, each hosting a subset of the components if beneficial orconvenient for an application, without changing the design.

The Object Manager 501 and the Storage Pool Managers 504 are softwarecomponents that may reside on the computer system platform thatinterconnects the storage resources and the computer systems that usethose storage resources, where the user's application resides. Theplacement of these software components on the interconnect platform isdesignated as a preferred embodiment, and may provide the ability toconnect customer systems to storage via communication protocols widelyused for such applications (e.g. Fibre Channel, iSCSI, etc.), and mayalso provide ease of deployment of the various software components.

The Object Manager 501 and Storage Pool Manager 504 communicate with theunderlying storage virtualization platform via the ApplicationProgramming Interfaces made available by the platform. These interfacesallow the software components to query and control the behavior of thecomputer system and how it interconnects the storage resources and thecomputer system where the user's Application resides. The componentsapply modularity techniques as is common within the practice to allowreplacement of the intercommunication code particular to a givenplatform.

The Object Manager and Storage Pool Managers communicate via a protocol.These are transmitted over standard networking protocols, e.g. TCP/IP,or standard Interprocess Communication (IPC) mechanisms typicallyavailable on the computer system. This allows comparable communicationbetween the components if they reside on the same computer platform oron multiple computer platforms connected by a network, depending on theparticular computer platform. The current configuration has all of thelocal software components residing on the same computer system for easeof deployment. This is not a strict requirement of the design, asdescribed above, and can be reconfigured in the future as needed.

Object Manager

Object Manager 501 is a software component for maintaining Data StorageObjects, and provides a set of protocol operations to control it. Theoperations include creation, destruction, duplication, and copying ofdata among the objects, maintaining access to objects, and in particularallow the specification of the storage pool used to create copies. Thereis no common subset of functions supported by all pools; however, in apreferred embodiment, primary pools may be performance-optimized, i.e.lower latency, whereas backup or replication pools may becapacity-optimized, supporting larger quantities of data andcontent-addressable. The pools may be remote or local. The storage poolsare classified according to various criteria, including means by which auser may make a business decision, e.g. cost per gigabyte of storage.

First, the particular storage device from which the storage is drawn maybe a consideration, as equipment is allocated for different businesspurposes, along with associated cost and other practical considerations.Some devices may not even be actual hardware but capacity provided as aservice, and selection of such a resource can be done for practicalbusiness purposes.

Second, the network topological “proximity” is considered, as nearstorage is typically connected by low-latency, inexpensive networkresources, while distant storage may be connected by high-latency,bandwidth limited expensive network resources; conversely, the distanceof a storage pool relative to the source may be beneficial whengeographic diversity protects against a physical disaster affectinglocal resources.

Third, storage optimization characteristics are considered, where somestorage is optimized for space-efficient storage, but requirescomputation time and resources to analyze or transform the data beforeit can be stored, while other storage by comparison is “performanceoptimized,” taking more storage resources by comparison but usingcomparatively little computation time or resource to transform the data,if at all.

Fourth, “speed of access” characteristics are considered, where someresources intrinsic to a storage computer platform are readily andquickly made available to the user's Application, e.g. as a virtual SCSIblock device, while some can only be indirectly used. These ease andspeed of recovery is often governed by the kind of storage used, andthis allows it to be suitably classified.

Fifth, the amount of storage used and the amount available in a givenpool are considered, as there may be benefit to either concentrating orspreading the storage capacity used.

The Service Level Policy Engine, described below, combines the SLAprovided by the user with the classification criteria to determine howand when to maintain the application data, and from which storage poolsto draw the needed resources to meet the Service Level Agreement (SLA).

The object manager 501 creates, maintains and employs a historymechanism to track the series of operations performed on a data objectwithin the performance pools, and to correlate those operations withothers that move the object to other storage pools, in particularcapacity-optimized ones. This series of records for each data object ismaintained at the object manager for all data objects in the primarypool, initially correlated by primary data object, then correlated byoperation order: a time line for each object and a list of all such timelines. Each operation performed exploits underlying virtualizationprimitives to capture the state of the data object at a given point intime.

Additionally, the underlying storage virtualization appliance may bemodified to expose and allow retrieval of internal data structures, suchas bitmaps, that indicate the modification of portions of the datawithin the data object. These data structures are exploited to capturethe state of a data object at a point in time: e.g., a snapshot of thedata object, and to provide differences between snapshots taken at aspecific time, and thereby enables optimal backup and restore. While theparticular implementations and data structures may vary among differentappliances from different vendors, a data structure is employed to trackchanges to the data object, and storage is employed to retain theoriginal state of those portions of the object that have changed:indications in the data structure correspond to data retained in thestorage. When accessing the snapshot, the data structure is consultedand for portions that have been changed, the preserved data is accessedrather than the current data, as the data object has been modified atthe areas so indicated. A typical data structure employed is a bitmap,where each bit corresponds to a section of the data object. Setting thebit indicates that section has been modified after the point in time ofthe snapshot operation. The underlying snapshot primitive mechanismmaintains this for as long as the snapshot object exists.

The time line described above maintains a list of the snapshotoperations against a given primary data object, including the time anoperation is started, the time it is stopped (if at all), a reference tothe snapshot object, and a reference to the internal data structure(e.g. bitmaps or extent lists), so that it can be obtained from theunderlying system. Also maintained is a reference to the result ofcopying the state of the data object at any given point in time intoanother pool—as an example, copying the state of a data object into acapacity—optimized pool 407 using content addressing results in anobject handle. That object handle corresponds to a given snapshot and isstored with the snapshot operation in the time line. This correlation isused to identify suitable starting points.

Optimal backup and restore consult the list of operations from a desiredstarting point to an end point. A time ordered list of operations andtheir corresponding data structures (bitmaps) are constructed such thata continuous time series from start to finish is realized: there is nogap between start times of the operations in the series. This ensuresthat all changes to the data object are represented by the correspondingbitmap data structures. It is not necessary to retrieve all operationsfrom start to finish; simultaneously existing data objects andunderlying snapshots overlap in time; it is only necessary that thereare no gaps in time where a change might have occurred that was nottracked. As bitmaps indicate that a certain block of storage has changedbut not what the change is, the bitmaps may be added or composedtogether to realize a set of all changes that occurred in the timeinterval. Instead of using this data structure to access the state at apoint in time, the system instead exploits the fact that the datastructure represents data modified as time marches forward. Rather, theend state of the data object is accessed at the indicated areas, thusreturning the set of changes to the given data object from the givenstart time to the end time.

The backup operation exploits this time line, the correlated references,and access to the internal data structures to realize our backupoperation. Similarly, it uses the system in a complementary fashion toaccomplish our restore operation. The specific steps are described belowin the section for “Optimal Backup/Restore.”

Virtual Storage Pool Types

FIG. 5 illustrates several representative storage pool types. Althoughone primary storage pool and two secondary storage pools are depicted inthe figure, many more may be configured in some embodiments.

Primary Storage Pool 507—contains the storage resources used to createthe data objects in which the user Application stores its data. This isin contrast to the other storage pools, which exist to primarily fulfillthe operation of the Data Management Virtualization Engine.

Performance Optimized Pool 508—a virtual storage pool able to providehigh performance backup (i.e. point in time duplication, describedbelow) as well as rapid access to the backup image by the userApplication

Capacity Optimized Pool 509—a virtual storage pool that chiefly providesstorage of a data object in a highly space-efficient manner by use ofdeduplication techniques described below. The virtual storage poolprovides access to the copy of the data object, but does not do so withhigh performance as its chief aim, in contrast to the PerformanceOptimized pool above.

The initial deployments contain storage pools as described above, as aminimal operational set. The design fully expects multiple Pools of avariety of types, representing various combinations of the criteriaillustrated above, and multiple Pool Managers as is convenient torepresent all of the storage in future deployments. The tradeoffsillustrated above are typical of computer data storage systems.

From a practical point of view, these three pools represent a preferredembodiment, addressing most users requirements in a very simple way.Most users will find that if they have one pool of storage for urgentrestore needs, which affords quick recovery, and one other pool that islow cost, so that a large number of images can be retained for a largeperiod of time, almost all of the business requirements for dataprotection can be met with little compromise.

The format of data in each pool is dictated by the objectives andtechnology used within the pool. For example, the quick recovery pool ismaintained in the form very similar to the original data to minimize thetranslation required and to improve the speed of recovery. The long-termstorage pool, on the other hand, uses deduplication and compression toreduce the size of the data and thus reduce the cost of storage.

Object Management Operations 505

The Object Manager 501 creates and maintains instances of Data StorageObjects 503 from the Virtual Storage Pools 418 according to theinstructions sent to it by the Service Level Policy Engine 406. TheObject Manager provides data object operations in five major areas:point-in-time duplication or copying (commonly referred to as“snapshots”), standard copying, object maintenance, mapping and accessmaintenance, and collections.

Object Management operations also include a series of Resource Discoveryoperations for maintaining Virtual Storage Pools themselves andretrieving information about them. The Pool Manager 504 ultimatelysupplies the functionality for these.

Point-In-Time Copy (“Snapshot”) Operations

Snapshot operations create a data object instance representing aninitial object instance at a specific point in time. More specifically,a snapshot operation creates a complete virtual copy of the members of acollection using the resources of a specified Virtual Storage Pool. Thisis called a Data Storage Object. Multiple states of a Data StorageObject are maintained over time, such that the state of a Data StorageObject as it existed at a point in time is available. As describedabove, a virtual copy is a copy implemented using an underlying storagevirtualization API that allows a copy to be created in a lightweightfashion, using copy-on-write or other in-band technologies instead ofcopying and storing all bits of duplicate data to disk. This may beimplemented using software modules written to access the capabilities ofan off-the-shelf underlying storage virtualization system such asprovided by EMC, vmware or IBM in some embodiments. Where suchunderlying virtualizations are not available, the described system mayprovide its own virtualization layer for interfacing with unintelligenthardware.

Snapshot operations require the application to freeze the state of thedata to a specific point so that the image data is coherent, and so thatthe snapshot may later be used to restore the state of the applicationat the time of the snapshot. Other preparatory steps may also berequired. These are handled by the Application-Specific Module 302,which is described in a subsequent section. For live applications,therefore, the most lightweight operations are desired.

Snapshot operations are used as the data primitive for all higher-leveloperations in the system. In effect, they provide access to the state ofthe data at a particular point in time. As well, since snapshots aretypically implemented using copy-on-write techniques that distinguishwhat has changed from what is resident on disk, these snapshots providedifferences that can also be composed or added together to efficientlycopy data throughout the system. The format of the snapshot may be theformat of data that is copied by Data Mover 502, which is describedbelow.

Standard Copy Operations

When a copy operation is not a snapshot, it may be considered a standardcopy operation. A standard copy operation copies all or a subset of asource data object in one storage pool to a data object in anotherstorage pool. The result is two distinct objects. One type of standardcopy operation that may be used is an initial “baseline” copy. This istypically done when data is initially copied from one Virtual StoragePool into another, such as from a performance-optimized pool to acapacity-optimized storage pool. Another type of standard copy operationmay be used wherein only changed data or differences are copied to atarget storage pool to update the target object. This would occur afteran initial baseline copy has previously been performed.

A complete exhaustive version of an object need not be preserved in thesystem each time a copy is made, even though a baseline copy is neededwhen the Data Virtualization System is first initialized. This isbecause each virtual copy provides access to a complete copy. Any deltaor difference can be expressed in relation to a virtual copy instead ofin relation to a baseline. This has the positive side effect ofvirtually eliminating the common step of walking through a series ofchange lists.

Standard copy operations are initiated by a series of instructions orrequests supplied by the Pool Manager and received by the Data Mover tocause the movement of data among the Data Storage Objects, and tomaintain the Data Storage Objects themselves. The copy operations allowthe creation of copies of the specified Data Storage Objects using theresources of a specified Virtual Storage Pool. The result is a copy ofthe source Data Object in a target Data Object in the storage pool.

The Snapshot and Copy operations are each structured with a preparationoperation and an activation operation. The two steps of prepare andactivate allow the long-running resource allocation operations, typicalof the prepare phase, to be decoupled from the actuation. This isrequired by applications that can only be paused for a short while tofulfill the point-in-time characteristics of a snapshot operation, whichin reality takes a finite but non-zero amount of time to accomplish.Similarly for copy and snapshot operations, this two-step preparationand activation structure allows the Policy Engine to proceed with anoperation only if resources for all of the collection members can beallocated.

Object Maintenance

Object Maintenance operations are a series of operations for maintainingdata objects, including creation, destruction, and duplication. TheObject Manager and Data Mover use functionality provided by a PoolRequest Broker (more below) to implement these operations. The dataobjects may be maintained at a global level, at each Storage Pool, orpreferably both.

Collections

Collection operations are auxiliary functions. Collections are abstractsoftware concepts, lists maintained in memory by the object manager.They allow the Policy Engine 206 to request a series of operations overall of the members in a collection, allowing a consistent application ofa request to all members. The use of collections allows for simultaneousactivation of the point-in-time snapshot so that multiple Data StorageObjects are all captured at precisely the same point in time, as this istypically required by the application for a logically correct restore.The use of collections allows for convenient request of a copy operationacross all members of a collection, where an application would usemultiple storage objects as a logical whole.

Resource Discovery Operations

The Object Manager discovers Virtual Storage Pools by issuing ObjectManagement Operations 505 to the Pool Manager 504, and uses theinformation obtained about each of the pools to select one that meetsthe required criteria for a given request, or in the case where nonematch, a default pool is selected, and the Object Manager can thencreate a data storage object using resources from the selected VirtualStorage Pool.

Mapping and Access

The Object Manager also provides sets of Object Management operations toallow and maintain the availability of these objects to externalApplications. The first set is operations for registering andunregistering the computers where the user's Applications reside. Thecomputers are registered by the identities typical to the storagenetwork in use (e.g. Fibre Channel WWPN, iSCSI identity, etc.). Thesecond set is “mapping” operations, and when permitted by the storagepool from which an object is created, the Data Storage Object can be“mapped,” that is, made available for use to a computer on which a userApplication resides.

This availability takes a form appropriate to the storage, e.g. a blockdevice presented on a SAN as a Fibre Channel disk or iSCSI device on anetwork, a filesystem on a file sharing network, etc. and is usable bythe operating system on the Application computer. Similarly, an“unmapping” operation reverses the availability of the virtual storagedevice on the network to a user Application. In this way, data storedfor one Application, i.e. a backup, can be made available to anotherApplication on another computer at a later time, i.e. a restore.

502 Data Mover

The Data Mover 502 is a software component within the Object Manager andData Mover that reads and writes data among the various Data StorageObjects 503 according to instructions received from the Object Managerfor Snapshot (Point in Time) Copy requests and standard copy requests.The Data Mover provides operations for reading and writing data amonginstances of data objects throughout the system. The Data Mover alsoprovides operations that allow querying and maintaining the state oflong running operations that the Object Manager has requested for it toperform.

The Data Mover uses functionality from the Pool Functionality Providers(see FIG. 6) to accomplish its operation. The Snapshot functionalityprovider 608 allows creation of a data object instance representing aninitial object instance at a specific point in time. The DifferenceEngine functionality provider 614 is used to request a description ofthe differences between two data objects that are related in a temporalchain. For data objects stored on content-addressable pools, a specialfunctionality is provided that can provide differences between any twoarbitrary data objects. This functionality is also provided forperformance-optimized pools, in some cases by an underlying storagevirtualization system, and in other cases by a module that implementsthis on top of commodity storage. The Data Mover 502 uses theinformation about the differences to select the set of data that itcopies between instances of data objects 503.

For a given Pool, the Difference Engine Provider provides a specificrepresentation of the differences between two states of a Data StorageObject over time. For a Snapshot provider the changes between two pointsin time are recorded as writes to a given part of the Data StorageObject. In one embodiment, the difference is represented as a bitmapwhere each bit corresponds to an ordered list of the Data Object areas,starting at the first and ascending in order to the last, where a setbit indicates a modified area. This bitmap is derived from thecopy-on-write bitmaps used by the underlying storage virtualizationsystem. In another embodiment, the difference may be represented as alist of extents corresponding to changed areas of data. For a ContentAddressable storage provider 610, the representation is described below,and is used to determine efficiently the parts of two ContentAddressable Data Objects that differ.

The Data Mover uses this information to copy only those sections thatdiffer, so that a new version of a Data Object can be created from anexisting version by first duplicating it, obtaining the list ofdifferences, and then moving only the data corresponding to thosedifferences in the list. The Data Mover 502 traverses the list ofdifferences, moving the indicated areas from the source Data Object tothe target Data Object. (See Optimal Way for Data Backup and Restore.)

506 Copy Operation—Request Translation and Instructions

The Object Manager 501 instructs the Data Mover 502 through a series ofoperations to copy data among the data objects in the Virtual StoragePools 418. The procedure includes the following steps, starting at thereception of instructions:

First, create Collection request. A name for the collection is returned.

Second, add Object to Collection. The collection name from above is usedas well as the name of the source Data Object that is to be copied andthe name of two antecedents: a Data Object against which differences areto be taken in the source Storage Resource Pool, and a correspondingData Object in the target Storage Resource Pool. This step is repeatedfor each source Data Object to be operated on in this set.

Third, prepare Copy Request. The collection name is supplied as well asa Storage Resource Pool to act as a target. The prepare commandinstructs the Object Manager to contact the Storage Pool Manager tocreate the necessary target Data Objects, corresponding to each of thesources in the collection. The prepare command also supplies thecorresponding Data Object in the target Storage Resource Pool to beduplicated, so the Provider can duplicate the provided object and usethat as a target object. A reference name for the copy request isreturned.

Fourth, activate Copy Request. The reference name for the copy requestreturned above is supplied. The Data Mover is instructed to copy a givensource object to its corresponding target object. Each request includesa reference name as well as a sequence number to describe the overalljob (the entire set of source target pairs) as well as a sequence numberto describe each individual source-target pair. In addition to thesource-target pair, the names of the corresponding antecedents aresupplied as part of the Copy instruction.

Fifth, the Copy Engine uses the name of the Data Object in the sourcepool to obtain the differences between the antecedent and the sourcefrom the Difference Engine at the source. The indicated differences arethen transmitted from the source to the target. In one embodiment, thesedifferences are transmitted as bitmaps and data. In another embodiment,these differences are transmitted as extent lists and data.

503 Data Storage Objects

Data Storage Objects are software constructs that permit the storage andretrieval of Application data using idioms and methods familiar tocomputer data processing equipment and software. In practice thesecurrently take the form of a SCSI block device on a storage network,e.g. a SCSI LUN, or a content-addressable container, where a designatorfor the content is constructed from and uniquely identifies the datatherein. Data Storage Objects are created and maintained by issuinginstructions to the Pool Manager. The actual storage for persisting theApplication data is drawn from the Virtual Storage Pool from which theData Storage Object is created.

The structure of the data storage object varies depending on the storagepool from which it is created. For the objects that take the form of ablock device on a storage network, the data structure for a given blockdevice Data Object implements a mapping between the Logical BlockAddress (LBA) of each of the blocks within the Data Object to the deviceidentifier and LBA of the actual storage location. The identifier of theData Object is used to identify the set of mappings to be used. Thecurrent embodiment relies on the services provided by the underlyingphysical computer platform to implement this mapping, and relies on itsinternal data structures, such as bitmaps or extent lists.

For objects that take the form of a Content Addressable Container, thecontent signature is used as the identifier, and the Data Object isstored as is described below in the section about deduplication.

504 Pool Manager

A Pool Manager 504 is a software component for managing virtual storageresources and the associated functionality and characteristics asdescribed below. The Object manager 501 and Data Movement Engine 502communicate with one or more Pool Managers 504 to maintain Data StorageObjects 503.

510 Virtual Storage Resources

Virtual Storage Resources 510 are various kinds of storage madeavailable to the Pool Manager for implementing storage pool functions,as described below. In this embodiment, a storage virtualizer is used topresent various external Fibre Channel or iSCSI storage LUNs asvirtualized storage to the Pool Manager 504.

The Storage Pool Manager

FIG. 6 further illustrates the Storage Pool Manager 504. The purpose ofthe storage pool manager is to present underlying virtual storageresources to the Object Manager/Data Mover as Storage Resource Pools,which are abstractions of storage and data management functionality withcommon interfaces that are utilized by other components of the system.These common interfaces typically include a mechanism for identifyingand addressing data objects associated with a specific temporal state,and a mechanism for producing differences between data objects in theform of bitmaps or extents. In this embodiment, the pool managerpresents a Primary Storage Pool, a Performance Optimized Pool, and aCapacity Optimized Pool. The common interfaces allow the object managerto create and delete Data Storage objects in these pools, either ascopies of other data storage objects or as new objects, and the datamover can move data between data storage objects, and can use theresults of data object differencing operations.

The storage pool manager has a typical architecture for implementing acommon interface to diverse implementations of similar functionality,where some functionality is provided by “smart” underlying resources,and other functionality must be implemented on top of less functionalunderlying resources.

Pool request broker 602 and pool functionality providers 604 aresoftware modules executing in either the same process as the ObjectManager/Data Mover, or in another process communicating via a local ornetwork protocol such as TCP. In this embodiment the providers include aPrimary Storage provider 606, Snapshot provider 608, Content Addressableprovider 610, and Difference Engine provider 614, and these are furtherdescribed below. In another embodiment the set of providers may be asuperset of those shown here.

Virtual Storage Resources 510 are the different kinds of storage madeavailable to the Pool Manager for implementing storage pool functions.In this embodiment, the virtual storage resources include sets of SCSIlogical units from a storage virtualization system that runs on the samehardware as the pool manager, and accessible (for both data andmanagement operations) through a programmatic interface: in addition tostandard block storage functionality additional capabilities areavailable including creating and deleting snapshots, and trackingchanged portions of volumes. In another embodiment the virtual resourcescan be from an external storage system that exposes similarcapabilities, or may differ in interface (for example accessed through afile-system, or through a network interface such as CIFS, iSCSI orCDMI), in capability (for example, whether the resource supports anoperation to make a copy-on-write snapshot), or in non-functionalaspects (for example, high-speed/limited-capacity such as Solid StateDisk versus low-speed/high-capacity such as SATA disk). The capabilitiesand interface available determine which providers can consume thevirtual storage resources, and which pool functionality needs to beimplemented within the pool manager by one or more providers: forexample, this implementation of a content addressable storage provideronly requires “dumb” storage, and the implementation is entirely withincontent addressable provider 610; an underlying content addressablevirtual storage resource could be used instead with a simpler“pass-through” provider. Conversely, this implementation of a snapshotprovider is mostly “pass-through” and requires storage that exposes aquick point-in-time copy operation.

Pool Request Broker 602 is a simple software component that servicesrequests for storage pool specific functions by executing an appropriateset of pool functionality providers against the configured virtualstorage resource 510. The requests that can be serviced include, but arenot limited to, creating an object in a pool; deleting an object from apool; writing data to an object; reading data from an object; copying anobject within a pool; copying an object between pools; requesting asummary of the differences between two objects in a pool.

Primary storage provider 606 enables management interfaces (for example,creating and deleting snapshots, and tracking changed portions of files)to a virtual storage resource that is also exposed directly toapplications via an interface such as fibre channel, iSCSI, NFS or CIFS.

Snapshot provider 608 implements the function of making a point-in-timecopy of data from a Primary resource pool. This creates the abstractionof another resource pool populated with snapshots. As implemented, thepoint-in-time copy is a copy-on-write snapshot of the object from theprimary resource pool, consuming a second virtual storage resource toaccommodate the copy-on-write copies, since this managementfunctionality is exposed by the virtual storage resources used forprimary storage and for the snapshot provider.

Difference engine provider 614 can satisfy a request for two objects ina pool to be compared that are connected in a temporal chain. Thedifference sections between the two objects are identified andsummarized in a provider-specific way, e.g. using bitmaps or extents.For example, the difference sections might be represented as a bitmapwhere each set bit denotes a fixed size region where the two objectsdiffer; or the differences might be represented procedurally as a seriesof function calls or callbacks.

Depending on the virtual storage resource on which the pool is based, oron other providers implementing the pool, a difference engine mayproduce a result efficiently in various ways. As implemented, adifference engine acting on a pool implemented via a snapshot provideruses the copy-on-write nature of the snapshot provider to track changesto objects that have had snapshots made. Consecutive snapshots of asingle changing primary object thus have a record of the differencesthat is stored alongside them by the snapshot provider, and thedifference engine for snapshot pools simply retrieves this record ofchange. Also as implemented, a difference engine acting on a poolimplemented via a Content Addressable provider uses the efficient treestructure (see below, FIG. 12) of the content addressable implementationto do rapid comparisons between objects on demand.

Content addressable provider 610 implements a write-once contentaddressable interface to the virtual storage resource it consumes. Itsatisfies read, write, duplicate and delete operations. Each written orcopied object is identified by a unique handle that is derived from itscontent. The content addressable provider is described further below(FIG. 11).

Pool Manager Operations

In operation, the pool request broker 502 accepts requests for datamanipulation operations such as copy, snapshot, or delete on a pool orobject. The request broker determines which provider code from pool 504to execute by looking at the name or reference to the pool or object.The broker then translates the incoming service request into a form thatcan be handled by the specific pool functionality provider, and invokesthe appropriate sequence of provider operations.

For example, an incoming request could ask to make a snapshot from avolume in a primary storage pool, into a snapshot pool. The incomingrequest identifies the object (volume) in the primary storage pool byname, and the combination of name and operation (snapshot) determinesthat the snapshot provider should be invoked which can makepoint-in-time snapshots from the primary pool using the underlyingsnapshot capability. This snapshot provider will translate the requestinto the exact form required by the native copy-on-write functionperformed by the underlying storage virtualization appliance, such asbitmaps or extents, and it will translate the result of the nativecopy-on-write function to a storage volume handle that can be returnedto the object manager and used in future requests to the pool manager.

Optimal Way for Data Backup Using the Object Manager and Data Mover

Optimal Way for Data Backup is a series of operations to make successiveversions of Application Data objects over time, while minimizing theamount of data that must be copied by using bitmaps, extents and othertemporal difference information stored at the Object Mover. It storesthe application data in a data storage object and associates with it themetadata that relates the various changes to the application data overtime, such that changes over time can be readily identified.

In a preferred embodiment, the procedure includes the following steps:

-   1. The mechanism provides an initial reference state, e.g. T0, of    the Application Data within a Data Storage Object.-   2. Subsequent instances (versions) are created on demand over time    of the Data Storage Object in a Virtual Storage Pool that has a    Difference Engine Provider.-   3. Each successive version, e.g. T4, T5, uses the Difference Engine    Provider for the Virtual Storage Pool to obtain the difference    between it and the instance created prior to it, so that T5 is    stored as a reference to T4 and a set of differences between T5 and    T4.-   4. The Copy Engine receives a request to copy data from one data    object (the source) to another data object (the destination).-   5. If the Virtual Storage Pool in which the destination object will    be created contains no other objects created from prior versions of    the source data object, then a new object is created in the    destination Virtual Storage Pool and the entire contents of the    source data object are copied to the destination object; the    procedure is complete. Otherwise the next steps are followed.-   6. If the Virtual Storage Pool in which the destination object is    created contains objects created from prior versions of the source    data object, a recently created prior version in the destination    Virtual Storage Pool is selected for which there exists a    corresponding prior version in the Virtual Storage Pool of the    source data object. For example, if a copy of T5 is initiated from a    snapshot pool, and an object created at time T3 is the most recent    version available at the target, T3 is selected as the prior    version.-   7. Construct a time-ordered list of the versions of the source data    object, beginning with an initial version identified in the previous    step, and ending with the source data object that is about to be    copied. In the above example, at the snapshot pool, all states of    the object are available, but only the states including and    following T3 are of interest: T3, T4, T5.-   8. Construct a corresponding list of the differences between each    successive version in the list such that all of the differences,    from the beginning version of the list to the end are represented.    Difference both, identify which portion of data has changed and    includes the new data for the corresponding time. This creates a set    of differences from the target version to the source version, e.g.    the difference between T3 and T5.-   9. Create the destination object by duplicating the prior version of    the object identified in Step 6 in the destination Virtual Storage    Pool, e.g. object T3 in the target store.-   10. Copy the set of differences identified in the list created in    Step 8 from the source data object to the destination object; the    procedure is complete.

Each data object within the destination Virtual Storage Pool iscomplete; that is, it represents the entire data object and allowsaccess to the all of the Application Data at the point in time withoutrequiring external reference to state or representations at other pointsin time. The object is accessible without replaying all deltas from abaseline state to the present state. Furthermore, the duplication ofinitial and subsequent versions of the data object in the destinationVirtual Storage Pool does not require exhaustive duplication of theApplication Data contents therein. Finally, to arrive at second andsubsequent states requires only the transmission of the changes trackedand maintained, as described above, without exhaustive traversal,transmission or replication of the contents of the data storage object.

Optimal Way for Data Restore Using the Object Manager and Data Mover

Intuitively, the operation of the Optimal Way for Data Restore is theconverse of the Optimal Way for Data Backup. The procedure to recreatethe desired state of a data object in a destination Virtual Storage Poolat a given point in time includes the following steps:

-   1. Identify a version of the data object in another Virtual Storage    Pool that has a Difference Engine Provider, corresponding to the    desired state to be recreated. This is the source data object in the    source Virtual Storage Pool.-   2. Identify a preceding version of the data object to be recreated    in the destination Virtual Storage Pool.-   3. If no version of the data object is identified in Step 2, then    create a new destination object in the destination Virtual Storage    Pool and copy the data from the source data object to the    destination data object. The procedure is complete. Otherwise,    proceed with the following steps.-   4. If a version of the data object is identified in Step 2, then    identify a data object in the source Virtual Storage Pool    corresponding to the data object identified in Step 2.-   5. If no data object is identified in Step 4, then create a new    destination object in the destination Virtual Storage Pool and copy    the data from the source data object to the destination data object.    The procedure is complete. Otherwise, proceed with the following    steps.-   6. Create a new destination data object in the Destination Virtual    Storage Pool by duplicating the data object identified in Step 2.-   7. Employ the Difference Engine Provider for the source Virtual    Storage Pool to obtain the set of differences between the data    object identified in Step 1 and the data object identified in Step    4.-   8. Copy the data identified by the list created in Step 7 from the    source data object to the destination data object. The procedure is    complete.

Access to the desired state is complete: it does not require externalreference to other containers or other states. Establishing the desiredstate given a reference state requires neither exhaustive traversal norexhaustive transmission, only the retrieved changes indicated by theprovided representations within the source Virtual Storage Pool.

The Service Level Agreement

FIG. 7 illustrates the Service Level Agreement. The Service LevelAgreement captures the detailed business requirements with respect tosecondary copies of the application data. In the simplest description,the business requirements define when and how often copies are created,how long they are retained and in what type of storage pools thesecopies reside. This simplistic description does not capture severalaspects of the business requirements. The frequency of copy creation fora given type of pool may not be uniform across all hours of the day oracross all days of a week. Certain hours of the day, or certain days ofa week or month may represent more (or less) critical periods in theapplication data, and thus may call for more (or less) frequent copies.Similarly, all copies of application data in a particular pool may notbe required to be retained for the same length of time. For example, acopy of the application data created at the end of monthly processingmay need to be retained for a longer period of time than a copy in thesame storage pool created in the middle of a month.

The Service Level Agreement 304 of certain embodiments has been designedto represent all of these complexities that exist in the businessrequirements. The Service Level Agreement has four primary parts: thename, the description, the housekeeping attributes and a collection ofService Level Policies. As mentioned above, there is one SLA perapplication.

The name attribute 701 allows each Service Level Agreement to have aunique name.

The description attribute 702 is where the user can assign a helpfuldescription for the Service Level Agreement.

The Service Level agreement also has a number of housekeeping attributes703 that enable it to be maintained and revised. These attributesinclude but are not limited to the owner's identity, the dates and timesof creation, modification and access, priority, enable/disable flags.

The Service Level Agreement also contains a plurality of Service LevelPolicies 705. Some Service level Agreements may have just a singleService Level Policy. More typically, a single SLA may contain tens ofpolicies.

Each Service Level Policy includes at least the following, in certainembodiments: the source storage pool location 706 and type 708; thetarget storage pool location 710 and type 712; the frequency for thecreation of copies 714, expressed as a period of time; the length ofretention of the copy 716, expressed as a period of time; the hours ofoperation 718 during the day for this particular Service Level Policy;and the days of the week, month or year 720 on which this Service LevelPolicy applies.

Each Service Level Policy specifies a source and target storage pool,and the frequency of copies of application data that are desired betweenthose storage pools. Furthermore, the Service Level Policy specifies itshours of operation and days on which it is applicable. Each ServiceLevel Policy is the representation of one single statement in thebusiness requirements for the protection of application data. Forexample, if a particular application has a business requirement for anarchive copy to be created each month after the monthly close andretained for three years, this might translate to a Service level Policythat requires a copy from the Local Backup Storage Pool into theLong-term Archive Storage Pool at midnight on the last day of the month,with a retention of three years.

All of the Service Level Policies with a particular combination ofsource and destination pool and location, say for example, sourcePrimary Storage pool and destination local Snapshot pool, when takentogether, specify the business requirements for creating copies intothat particular destination pool. Business requirements may dictate forexample that snapshot copies be created every hour during regularworking hours, but only once every four hours outside of these times.Two Service Level Policies with the same source and target storage poolswill effectively capture these requirements in a form that can be putinto practice by the Service Policy Engine.

This form of a Service Level Agreement allows the representation of theschedule of daily, weekly and monthly business activities, and thuscaptures business requirements for protecting and managing applicationdata much more accurately than traditional RPO and RPO based schemes. Byallowing hour of operation and days, weeks, and months of the year,scheduling can occur on a “calendar basis.”

Taken together, all of the Service Level Policies with one particularcombination of source and destinations, for example, “source: localprimary and destination: local performance optimized”, captures thenon-uniform data protection requirements for one type of storage. Asingle RPO number, on the other hand, forces a single uniform frequencyof data protection across all times of day and all days. For example, acombination of Service Level Policies may require a large number ofsnapshots to be preserved for a short time, such as 10 minutes, and alesser number of snapshots to be preserved for a longer time, such as 8hours; this allows a small amount of information that has beenaccidentally deleted can be reverted to a state not more than 10 minutesbefore, while still providing substantial data protection at longer timehorizons without requiring the storage overhead of storing all snapshotstaken every ten minutes. As another example, the backup data protectionfunction may be given one Policy that operates with one frequency duringthe work week, and another frequency during the weekend.

When Service Level Policies for all of the different classes of sourceand destination storage are included, the Service Level Agreement fullycaptures all of the data protection requirements for the entireapplication, including local snapshots, local long duration stores,off-site storage, archives, etc. A collection of policies within a SLAis capable of expressing when a given function should be performed, andis capable of expressing multiple data management functions that shouldbe performed on a given source of data.

Service Level Agreements are created and modified by the user through auser interface on a management workstation. These agreements areelectronic documents stored by the Service Policy Engine in a structuredSQL database or other repository that it manages. The policies areretrieved, electronically analyzed, and acted upon by the Service PolicyEngine through its normal scheduling algorithm as described below.

FIG. 8 illustrates the Application Specific Module 402. The ApplicationSpecific module runs close to the Application 300 (as described above),and interacts with the Application and its operating environment togather metadata and to query and control the Application as required fordata management operations.

The Application Specific Module interacts with various components of theapplication and its operating environment including Application ServiceProcesses and Daemons 801, Application Configuration Data 802, OperatingSystem Storage Services 803 (such as VSS and VDS on Windows), LogicalVolume Management and Filesystem Services 804, and Operating SystemDrivers and Modules 805.

The Application Specific Module performs these operations in response tocontrol commands from the Service Policy Engine 406. There are twopurposes for these interactions with the application: MetadataCollection and Application Consistency.

Metadata Collection is the process by which the Application SpecificModule collects metadata about the application. In some embodiments,metadata includes information such as: configuration parameters for theapplication; state and status of the application; control files andstartup/shutdown scripts for the application; location of the datafiles,journal and transaction logs for the application; and symbolic links,filesystem mount points, logical volume names, and other such entitiesthat can affect the access to application data.

Metadata is collected and saved along with application data and SLAinformation. This guarantees that each copy of application data withinthe system is self contained and includes all of the details required torebuild the application data.

Application Consistency is the set of actions that ensure that when acopy of the application data is created, the copy is valid, and can berestored into a valid instance of the application. This is critical whenthe business requirements dictate that the application be protectedwhile it is live, in its online, operational state. The application mayhave interdependent data relations within its data stores, and if theseare not copied in a consistent state will not provide a valid restorableimage.

The exact process of achieving application consistency varies fromapplication to application. Some applications have a simple flushcommand that forces cached data to disk. Some applications support a hotbackup mode where the application ensures that its operations arejournaled in a manner that guarantees consistency even as applicationdata is changing. Some applications require interactions with operatingsystem storage services such as VSS and VDS to ensure consistency. TheApplication Specific Module is purpose-built to work with a particularapplication and to ensure the consistency of that application. TheApplication Specific Module interacts with the underlying storagevirtualization device and the Object Manager to provide consistentsnapshots of application data.

For efficiency, the preferred embodiment of the Application SpecificModule 402 is to run on the same server as Application 300. This assuresthe minimum latency in the interactions with the application, andprovides access to storage services and filesystems on the applicationhost. The application host is typically considered primary storage,which is then snapshotted to a performance-optimized store.

In order to minimize interruption of a running application, includingminimizing preparatory steps, the Application Specific Module is onlytriggered to make a snapshot when access to application data is requiredat a specific time, and when a snapshot for that time does not existelsewhere in the system, as tracked by the Object Manager. By trackingwhich times snapshots have been made, the Object Manager is able tofulfill subsequent data requests from the performance-optimized datastore, including for satisfying multiple requests for backup andreplication which may issue from secondary, capacity-optimized pools.The Object Manager may be able to provide object handles to the snapshotin the performance-optimized store, and may direct theperformance-optimized store in a native format that is specific to theformat of the snapshot, which is dependent on the underlying storageappliance. In some embodiments this format may be application datacombined with one or more LUN bitmaps indicating which blocks havechanged; in other embodiments it may be specific extents. The formatused for data transfer is thus able to transfer only a delta ordifference between two snapshots using bitmaps or extents.

Metadata, such as the version number of the application, may also bestored for each application along with the snapshot. When a SLA policyis executed, application metadata is read and used for the policy. Thismetadata is stored along with the data objects. For each SLA,application metadata will only be read once during the lightweightsnapshot operation, and preparatory operations which occur at that timesuch as flushing caches will only be performed once during thelightweight snapshot operation, even though this copy of applicationdata along with its metadata may be used for multiple data managementfunctions.

The Service Policy Engine

FIG. 9 illustrates the Service Policy Engine 406. The Service PolicyEngine contains the Service Policy Scheduler 902, which examines all ofthe Service Level Agreements configured by the user and makes schedulingdecisions to satisfy Service Level Agreements. It relies on several datastores to capture information and persist it over time, including, insome embodiments, a SLA Store 904, where configured Service LevelAgreements are persisted and updated; a Resource Profile Store 906,storing Resource Profiles that provide a mapping between logical storagepool names and actual storage pools; Protection Catalog Store 908, whereinformation is cataloged about previous successful copies created invarious pools that have not yet expired; and centralized History Store910.

History Store 910 is where historical information about past activitiesis saved for the use of all data management applications, including thetimestamp, order and hierarchy of previous copies of each applicationinto various storage pools. For example, a snapshot copy from a primarydata store to a capacity-optimized data store that is initiated at 1P.M. and is scheduled to expire at 9 P.M. will be recorded in HistoryStore 910 in a temporal data store that also includes linked object datafor snapshots for the same source and target that have taken place at 11A.M. and 12 P.M.

These stores are managed by the Service Policy Engine. For example, whenthe user, through the Management workstation creates a Service LevelAgreement, or modifies one of the policies within it, it is the ServicePolicy Engine that persists this new SLA in its store, and reacts tothis modification by scheduling copies as dictated by the SLA.Similarly, when the Service Policy Engine successfully completes a datamovement job that results in a new copy of an application in a StoragePool, the Storage Policy Engine updates the History Store, so that thiscopy will be factored into future decisions.

The preferred embodiment of the various stores used by the ServicePolicy Engine is in the form of tables in a relational databasemanagement system in close proximity to the Service Policy Engine. Thisensures consistent transactional semantics when querying and updatingthe stores, and allows for flexibility in retrieving interdependentdata.

The scheduling algorithm for the Service Policy Scheduler 902 isillustrated in FIG. 10. When the Service Policy Scheduler decides itneeds to make a copy of application data from one storage pool toanother, it initiates a Data Movement Requestor and Monitor task, 912.These tasks are not recurring tasks and terminate when they arecompleted. Depending on the way that Service Level Policies arespecified, a plurality of these requestors might be operational at thesame time.

The Service Policy Scheduler considers the priorities of Service LevelAgreements when determining which additional tasks to undertake. Forexample, if one Service Level Agreement has a high priority because itspecifies the protection for a mission-critical application, whereasanother SLA has a lower priority because it specifies the protection fora test database, then the Service Policy Engine may choose to run onlythe protection for the mission-critical application, and may postpone oreven entirely skip the protection for the lower priority application.This is accomplished by the Service Policy Engine scheduling a higherpriority SLA ahead of a lower priority SLA. In the preferred embodiment,in such a situation, for auditing purposes, the Service Policy Enginewill also trigger a notification event to the management workstation.

The Policy Scheduling Algorithm

FIG. 10 illustrates the flowchart of the Policy Schedule Engine. ThePolicy Schedule Engine continuously cycles through all the SLAs defined.When it gets to the end of all of the SLAs, it sleeps for a short while,e.g. 10 seconds, and resumes looking through the SLAs again. Each SLAencapsulates the complete data protection business requirements for oneapplication; thus all of the SLAs represent all of the applications.

For each SLA, the schedule engine collects together all of the ServiceLevel Policies that have the same source pool and destination pool 1004the process state at 1000 and iterates to the next SLA in the set ofSLAs in 1002. Taken together, this subset of the Service Level Policiesrepresent all of the requirements for a copy from that source storagepool to that particular destination storage pool.

Among this subset of Service Level Policies, the Service PolicyScheduler discards the policies that are not applicable to today, or areoutside their hours of operation. Among the policies that are left, findthe policy that has the shortest frequency 1006, and based on thehistory data and in history store 910, the one with the longestretention that needs to be run next 1008.

Next, there are a series of checks 1010-1014 which rule out making a newcopy of application data at this time—because the new copy is not yetdue, because a copy is already in progress or because there is not newdata to copy. If any of these conditions apply, the Service PolicyScheduler moves to the next combination of source and destination pools1004. If none of these conditions apply, a new copy is initiated. Thecopy is executed as specified in the corresponding service level policywithin this SLA 1016.

Next, the Scheduler moves to the next Source and Destination poolcombination for the same Service Level agreement 1018. If there are nomore distinct combinations, the Scheduler moves on to the next ServiceLevel Agreement 1020.

After the Service Policy Scheduler has been through allsource/destination pool combinations of all Service Level Agreements, itpauses for a short period and then resumes the cycle.

A simple example system with a snapshot store and a backup store, withonly 2 policies defined, would interact with the Service PolicyScheduler as follows. Given two policies, one stating “backup everyhour, the backup to be kept for 4 hours” and another stating “backupevery 2 hours, the backup to be kept for 8 hours,” the result would be asingle snapshot taken each hour, the snapshots each being copied to thebackup store but retained a different amount of time at both thesnapshot store and the backup store. The “backup every 2 hours” policyis scheduled to go into effect at 12:00 P.M by the system administrator.

At 4:00 P.M., when the Service Policy Scheduler begins operating at step1000, it finds the two policies at step 1002. (Both policies applybecause a multiple of two hours has elapsed since 12:00 P.M.) There isonly one source and destination pool combination at step 1004. There aretwo frequencies at step 1006, and the system selects the 1-hourfrequency because it is shorter than the 2-hour frequency. There are twooperations with different retentions at step 1008, and the systemselects the operation with the 8-hour retention, as it has the longerretention value. Instead of one copy being made to satisfy the 4-hourrequirement and another copy being made to satisfy the 8-hourrequirement, the two requirements are coalesced into the longer 8-hourrequirement, and are satisfied by a single snapshot copy operation. Thesystem determines that a copy is due at step 1010, and checks therelevant objects at the History Store 910 to determine if the copy hasalready been made at the target (at step 912) and at the source (at step914). If these checks are passed, the system initiates the copy at step916, and in the process triggers a snapshot to be made and saved at thesnapshot store. The snapshot is then copied from the snapshot store tothe backup store. The system then goes to sleep 1022 and wakes up againafter a short period, such as 10 seconds. The result is a copy at thebackup store and a copy at the snapshot store, where every even-hoursnapshot lasts for 8 hours, and every odd-hour snapshot lasts 4 hours.The even-hour snapshots at the backup store and the snapshot store areboth tagged with the retention period of 8 hours, and will beautomatically deleted from the system by another process at that time.

Note that there is no reason to take two snapshots or make two backupcopies at 2 o'clock, even though both policies apply, because bothpolicies are satisfied by a single copy. Combining and coalescing thesesnapshots results in the reduction of unneeded operations., whileretaining the flexibility of multiple separate policies. As well, it maybe helpful to have two policies active at the same time for the sametarget with different retention. In the example given, there are morehourly copies kept than two-hour copies, resulting in more granularityfor restore at times that are closer to the present. For example, in theprevious system, if at 7:30 P.M. damage is discovered from earlier inthe afternoon, a backup will be available for every hour for the pastfour hours: 4, 5, 6, 7 P.M. As well, two more backups will have beenretained from 2 P.M. and 12 P.M.

The Content Addressable Store

FIG. 11 is a block diagram of the modules implementing the contentaddressable store for the Content Addressable Provider 510.

The content addressable store 510 implementation provides a storageresource pool that is optimized for capacity rather than for copy-in orcopy-out speed, as would be the case for the performance-optimized poolimplemented through snapshots, described earlier, and thus is typicallyused for offline backup, replication and remote backup. Contentaddressable storage provides a way of storing common subsets ofdifferent objects only once, where those common subsets may be ofvarying sizes but typically as small as 4 KiBytes. The storage overheadof a content addressable store is low compared to a snapshot store,though the access time is usually higher. Generally objects in a contentaddressable store have no intrinsic relationship to one another, eventhough they may share a large percentage of their content, though inthis implementation a history relationship is also maintained, which isan enabler of various optimizations to be described. This contrasts witha snapshot store where snapshots intrinsically form a chain, eachstoring just deltas from a previous snapshot or baseline copy. Inparticular, the content addressable store will store only one copy of adata subset that is repeated multiple times within a single object,whereas a snapshot-based store will store at least one full-copy of anyobject.

The content addressable store 510 is a software module that executes onthe same system as the pool manager, either in the same process or in aseparate process communicating via a local transport such as TCP. Inthis embodiment, the content addressable store module runs in a separateprocess so as to minimize impact of software failures from differentcomponents.

This module's purpose is to allow storage of Data Storage Objects 403 ina highly space-efficient manner by deduplicating content (i.e., ensuringrepeated content within single or multiple data objects is stored onlyonce).

The content addressable store module provides services to the poolmanager via a programmatic API. These services include the following:

Object to Handle mapping 1102: an object can be created by writing datainto the store via an API; once the data is written completely the APIreturns an object handle determined by the content of the object.Conversely, data may be read as a stream of bytes from an offset withinan object by providing the handle. Details of how the handle isconstructed are explained in connection with the description of FIG. 12.

Temporal Tree Management 1104 tracks parent/child relationships betweendata objects stored. When a data object is written into the store 510,an API allows it to be linked as a child to a parent object already inthe store. This indicates to the content addressable store that thechild object is a modification of the parent. A single parent may havemultiple children with different modifications, as might be the case forexample if an application's data were saved into the store regularly forsome while; then an early copy were restored and used as a new startingpoint for subsequent modifications. Temporal tree management operationsand data models are described in more detail below.

Difference Engine 1106 can generate a summary of difference regionsbetween two arbitrary objects in the store. The differencing operationis invoked via an API specifying the handles of two objects to becompared, and the form of the difference summary is a sequence ofcallbacks with the offset and size of sequential difference sections.The difference is calculated by comparing two hashed representations ofthe objects in parallel.

Garbage Collector 1108 is a service that analyzes the store to findsaved data that is not referenced by any object handle, and to reclaimthe storage space committed to this data. It is the nature of thecontent addressable store that much data is referenced by multipleobject handles, i.e., the data is shared between data objects; some datawill be referenced by a single object handle; but data that isreferenced by no object handles (as might be the case if an objecthandle has been deleted from the content addressable system) can besafely overwritten by new data.

Object Replicator 1110 is a service to duplicate data objects betweentwo different content addressable stores. Multiple content addressablestores may be used to satisfy additional business requirements, such asoffline backup or remote backup.

These services are implemented using the functional modules shown inFIG. 11. The Data Hash module 1112 generates fixed length keys for datachunks up to a fixed size limit. For example, in this embodiment themaximum size of chunk that the hash generator will make a key for is 64KiB. The fixed length key is either a hash, tagged to indicate thehashing scheme used, or a non-lossy algorithmic encoding. The hashingscheme used in this embodiment is SHA-1, which generates a securecryptographic hash with a uniform distribution and a probability of hashcollision near enough zero that no facility need be incorporated intothis system to detect and deal with collisions.

The Data Handle Cache 1114 is a software module managing an in-memorydatabase that provides ephemeral storage for data and for handle-to-datamappings.

The Persistent Handle Management Index 1104 is a reliable persistentdatabase of CAH-to-data mappings. In this embodiment it is implementedas a B-tree, mapping hashes from the hash generator to pages in thepersistent data store 1118 that contain the data for this hash. Sincethe full B-tree cannot be held in memory at one time, for efficiency,this embodiment also uses an in-memory bloom filter to avoid expensiveB-tree searches for hashes known not to be present.

The Persistent Data Storage module 1118 stores data and handles tolong-term persistent storage, returning a token indicating where thedata is stored. The handle/token pair is subsequently used to retrievethe data. As data is written to persistent storage, it passes through alayer of lossless data compression 1120, in this embodiment implementedusing zlib, and a layer of optional reversible encryption 1122, which isnot enabled in this embodiment.

For example, copying a data object into the content addressable store isan operation provided by the object/handle mapper service, since anincoming object will be stored and a handle will be returned to therequestor. The object/handle mapper reads the incoming object, requestshashes to be generated by the Data Hash Generator, stores the data toPersistent Data Storage and the handle to the Persistent HandleManagement Index. The Data Handle Cache is kept updated for future quicklookups of data for the handle. Data stored to Persistent Data Storageis compressed and (optionally) encrypted before being written to disk.Typically a request to copy in a data object will also invoke thetemporal tree management service to make a history record for theobject, and this is also persisted via Persistent Data Storage.

As another example, copying a data object out of the content addressablestore given its handle is another operation provided by theobject/handle mapper service. The handle is looked up in the Data HandleCache to locate the corresponding data; if the data is missing in thecache the persistent index is used; once the data is located on disk, itis retrieved via persistent data storage module (which decrypts anddecompresses the disk data) and then reconstituted to return to therequestor.

The Content Addressable Store Handle

FIG. 12 shows how the handle for a content addressed object isgenerated. The data object manager references all content addressableobjects with a content addressable handle. This handle is made up ofthree parts. The first part 1201 is the size of the underlying dataobject the handle immediately points to. The second part 1202 is thedepth of object it points to. The third 1203 is a hash of the object itpoints to. Field 1203 optionally includes a tag indicating that the hashis a non-lossy encoding of the underlying data. The tag indicates theencoding scheme used, such as a form of run-length encoding (RLE) ofdata used as an algorithmic encoding if the data chunk can be fullyrepresented as a short enough RLE. If the underlying data object is toolarge to be represented as a non-lossy encoding, a mapping from the hashto a pointer or reference to the data is stored separately in thepersistent handle management index 1104.

The data for a content addressable object is broken up into chunks 1204.The size of each chunk must be addressable by one content addressablehandle 1205. The data is hashed by the data hash module 1102, and thehash of the chunk is used to make the handle. If the data of the objectfits in one chunk, then the handle created is the final handle of theobject. If not, then the handles themselves are grouped together intochunks 1206 and a hash is generated for each group of handles. Thisgrouping of handles continues 1207 until there is only one handle 1208produced which is then the handle for the object.

When an object is to be reconstituted from a content handle (thecopy-out operation for the storage resource pool), the top level contenthandle is dereferenced to obtain a list of next-level content handles.These are dereferenced in turn to obtain further lists of contenthandles until depth-0 handles are obtained. These are expanded to data,either by looking up the handle in the handle management index or cache,or (in the case of an algorithmic hash such as run-length encoding)expanding deterministically to the full content.

Temporal Tree Management

FIG. 13 illustrates the temporal tree relationship created for dataobjects stored within the content addressable store. This particulardata structure is utilized only within the content addressable store.The temporal tree management module maintains data structures 1302 inthe persistent store that associate each content-addressed data objectto a parent (which may be null, to indicate the first in a sequence ofrevisions). The individual nodes of the tree contain a single hashvalue. This hash value references a chunk of data, if the hash is adepth-0 hash, or a list of other hashes, if the hash is a depth-1 orhigher hash. The references mapped to a hash value is contained in thePersistent Handle Management Index 1104. In some embodiments the edgesof the tree may have weights or lengths, which may be used in analgorithm for finding neighbors.

This is a standard tree data structure and the module supports standardmanipulation operations, in particular: 1310 Add: adding a leaf below aparent, which results in a change to the tree as between initial state1302 and after-add state 1304; and 1312 Remove: removing a node (andreparenting its children to its parent), which results in a change tothe tree as between after-add state 1304 and after-remove state 1306.

The “Add” operation may be used whenever an object is copied-in to theCAS from an external pool. If the copy-in is via the Optimal Way forData Backup, or if the object is originating in a different CAS pool,then it is required that a predecessor object be specified, and the Addoperation is invoked to record this predecessor/successor relationship.

The “Remove” operation is invoked by the object manager when the policymanager determines that an object's retention period has expired. Thismay lead to data stored in the CAS having no object in the temporal treereferring to it, and therefore a subsequent garbage collection pass canfree up the storage space for that data as available for re-use.

Note that it is possible for a single predecessor to have multiplesuccessors or child nodes. For example, this may occur if an object isoriginally created at time T1 and modified at time T2, the modificationsare rolled back via a restore operation, and subsequent modificationsare made at time T3. In this example, state T1 has two children, stateT2 and state T3.

Different CAS pools may be used to accomplish different businessobjectives such as providing disaster recovery in a remote location.When copying from one CAS to another CAS, the copy may be sent as hashesand offsets, to take advantage of the native deduplication capabilitiesof the target CAS. The underlying data pointed to by any new hashes isalso sent on an as-needed basis.

The temporal tree structure is read or navigated as part of theimplementation of various services:

Garbage Collection navigates the tree in order to reduce the cost of the“mark” phase, as described below

Replication to a different CAS pool finds a set of near-neighbors in thetemporal tree that are also known to have been transferred already tothe other CAS pool, so that only a small set of differences need to betransferred additionally

Optimal-Way for data restore uses the temporal tree to find apredecessor that can be used as a basis for the restore operation. Inthe CAS temporal tree data structure, children are subsequent versions,e.g., as dictated by archive policy. Multiple children are supported onthe same parent node; this case may arise when a parent node is changed,then used as the basis for a restore, and subsequently changed again.

CAS Difference Engine

The CAS difference engine 1106 compares two objects identified by hashvalues or handles as in FIGS. 11 and 12, and produces a sequence ofoffsets and extents within the objects where the object data is known todiffer. This sequence is achieved by traversing the two object trees inparallel in the hash data structure of FIG. 12. The tree traversal is astandard depth- or breadth-first traversal. During traversal, the hashesat the current depth are compared. Where the hash of a node is identicalbetween both sides, there is no need to descend the tree further, so thetraversal may be pruned. If the hash of a node is not identical, thetraversal continues descending into the next lowest level of the tree.If the traversal reaches a depth-0 hash that is not identical to itscounterpart, then the absolute offset into the data object beingcompared where the non-identical data occurs, together with the datalength, is emitted into the output sequence. If one object is smaller insize than another, then its traversal will complete earlier, and allsubsequent offsets encountered in the traversal of the other are emittedas differences.

Garbage Collection via Differencing

As described under FIG. 11, Garbage Collector is a service that analyzesa particular CAS store to find saved data that is not referenced by anyobject handle in the CAS store temporal data structure, and to reclaimthe storage space committed to this data. Garbage collection uses astandard “Mark and Sweep” approach. Since the “mark” phase may be quiteexpensive, the algorithm used for the mark phase attempts to minimizemarking the same data multiple times, even though it may be referencedmany times; however the mark phase must be complete, ensuring that noreferenced data is left unmarked, as this would result in data loss fromthe store as, after a sweep phase, unmarked data would later beoverwritten by new data.

The algorithm employed for marking referenced data uses the fact thatobjects in the CAS are arranged in graphs with temporal relationshipsusing the data structure depicted in FIG. 13. It is likely that objectsthat share an edge in these graphs differ in only a small subset oftheir data, and it is also rare that any new data chunk that appearswhen an object is created from a predecessor should appear again betweenany two other objects. Thus, the mark phase of garbage collectionprocesses each connected component of the temporal graph.

FIG. 14 is an example of garbage collection using temporal relationshipsin certain embodiments. A depth-first search is made, represented byarrows 1402, of a data structure containing temporal relationships. Takea starting node 1404 from which to begin the tree traversal. Node 1404is the tree root and references no objects. Node 1406 containsreferences to objects H1 and H2, denoting a hash value for object 1 anda hash value for object 2. All depth-0, depth-1 and higher data objectsthat are referenced by node 1406, here H1 and H2, are enumerated andmarked as referenced.

Next, node 1408 is processed. As it shares an edge with node 1406, whichhas been marked, the difference engine is applied to the differencebetween the object referenced by 1406 and the object referenced by 1408,obtaining a set of depth-0, depth-1 and higher hashes that exist in theunmarked object but not in the marked object. In the figure, the hashthat exists in node 1408 but not in node 1406 is H3, so H3 is marked asreferenced. This procedure is continued until all edges are exhausted.

A comparison of the results produced by a prior art algorithm 1418 andthe present embodiment 1420 shows that when node 1408 is processed bythe prior art algorithm, previously-seen hashes H1 and H2 are emittedinto the output stream along with new hash H3. Present embodiment 1420does not emit previously seen hashes into the output stream, resultingin only new hashes H3, H4, H5, H6, H7 being emitted into the outputstream, with a corresponding improvement in performance. Note that thismethod does not guarantee that data will not be marked more than once.For example, if hash value H4 occurs independently in node 1416, it willbe independently marked a second time.

Copy an Object into the CAS

Copying an object from another pool into the CAS uses the softwaremodules described in FIG. 11 to produce a data structure referenced byan object handle as in FIG. 12. The input to the process is (a) asequence of chunks of data at specified offsets, sized appropriately formaking depth-0 handles, and optionally (b) a previous version of thesame object. Implicitly, the new object will be identical to theprevious version except where the input data is provided and itselfdiffers from the previous version. The algorithm for the copy-inoperation is illustrated in a flowchart at FIG. 15.

If a previous version (b) is provided, then the sequence (a) may be asparse set of changes from (b). In the case that the object to be copiedand is known to differ from a previous object at only a few points, thiscan greatly reduce the amount of data that needs to be copied in, andtherefore reduce the computation and i/o activity required. This is thecase, for example, when the object is to be copied in via the optimalway for data backup described previously.

Even if the sequence (a) includes sections that are largely unchangedfrom a predecessor, identifying the predecessor (b) allows the copy-inprocedure to do quick checks as to whether the data has indeed changedand therefore to avoid data duplication at a finer level of granularitythan might be possible for the difference engine in some other storagepool providing input to a CAS.

Implicitly then, the new object will be identical to the previousversion except where the input data is provided and itself differs fromthe previous version. The algorithm for the copy-in operation isillustrated in a flowchart at FIG. 15.

The process starts at step 1500 as an arbitrarily-sized data object inthe temporal store is provided, and proceeds to 1502, which enumeratesany and all hashes (depth-0 through the highest level) referenced by thehash value in the predecessor object, if such is provided. This will beused as a quick check to avoid storing data that is already contained inthe predecessor.

At step 1504, if a predecessor is input, create a reference to a cloneof it in the content-addressable data store temporal data structure.This clone will be updated to become the new object. Thus the new objectwill become a copy of the predecessor modified by the differences copiedinto the CAS from the copying source pool.

At steps 1506, 1508, the Data Mover 502 pushes the data into the CAS.The data is accompanied by an object reference and an offset, which isthe target location for the data. The data may be sparse, as only thedifferences from the predecessor need to be moved into the new object.At this point the incoming data is broken into depth-0 chunks sizedsmall enough that each can be represented by a single depth-0 hash.

At step 1510, the data hash module generates a hash for each depth-0chunk.

At step 1512, read the predecessor hash at the same offset. If the hashof the data matches the hash of the predecessor at the same offset, thenno data needs to be stored and the depth-1 and higher objects do notneed to be updated for this depth-0 chunk. In this case, return toaccept the next depth-0 chunk of data. This achieves temporaldeduplication without having to do expensive global lookups. Even thoughthe source system is ideally sending only the differences from the datathat has previously been stored in the CAS, this check may be necessaryif the source system is performing differencing at a different level ofgranularity, or if the data is marked as changed but has been changedback to its previously-stored value. Differencing may be performed at adifferent level of granularity if, for example, the source system is asnapshot pool which creates deltas on a 32 KiB boundary and the CASstore creates hashes on 4 KiB chunks.

If a match is not found, the data may be hashed and stored. Data iswritten starting at the provided offset and ending once the new data hasbeen exhausted. Once the data has been stored, at step 1516, if theoffset is still contained within the same depth-1 object, then depth-1,depth-2 and all higher objects 1518 are updated, generating new hashesat each level, and the depth-0, depth-1 and all higher objects arestored at step 1514 to a local cache.

However, at step 1520, if the amount of data to be stored exceeds thedepth-1 chunk size and the offset is to be contained in a new depth-1object, the current depth-1 must be flushed to the store, unless it isdetermined to be stored there already. First look it up in the globalindex 1116. If it is found there, remove the depth-1 and all associateddepth-0 objects from the local cache and proceed with the new chunk1522.

At step 1524, as a quick check to avoid visiting the global index, foreach depth-0, depth-1 and higher object in the local cache, lookup itshash in the local store established in 1502. Discard any that match.

At step 1526, for each depth-0, depth-1 and higher object in the localcache, lookup its hash in the global index 1116. Discard any that match.This ensures that data is deduplicated globally.

At step 1528: store all remaining content from the local cache into thepersistent store, then continue to process the new chunk.

Reading an object out of the CAS is a simpler process and is commonacross many implementations of CAS. The handle for the object is mappedto a persistent data object via the global index, and the offsetrequired is read from within this persistent data. In some cases it maybe necessary to recurse through several depths in the object handletree.

CAS Object Network Replication

As described under FIG. 11, the Replicator 1110 is a service toduplicate data objects between two different content addressable stores.The process of replication could be achieved through reading out of onestore and writing back into another, but this architecture allows moreefficient replication over a limited bandwidth connection such as alocal- or wide-area network.

A replicating system operating on each CAS store uses the differenceengine service described above together with the temporal relationshipstructure as described in FIG. 13, and additionally stores on aper-object basis in the temporal data structure used by the CAS store arecord of what remote store the object has been replicated to. Thisprovides definitive knowledge of object presence at a certain datastore.

Using the temporal data structure, it is possible for the system todetermine which objects exist on which data stores. This information isleveraged by the Data Mover and Difference Engine to determine a minimalsubset of data to be sent over the network during a copy operation tobring a target data store up to date. For example, if data object O hasbeen copied at time T3 from a server in Boston to a remote server inSeattle, Protection Catalog Store 908 will store that object O at timeT3 exists both in Boston and Seattle. At time T5, during a subsequentcopy from Boston to Seattle, the temporal data structure will beconsulted to determine the previous state of object O in Seattle thatshould be used for differencing on the source server in Boston. TheBoston server will then take the difference of T5 and T3, and send thatdifference to the Seattle server.

The process to replicate an object A is then as follows: Identify anobject A0 that is recorded as having already been replicated to thetarget store and a near neighbor of A in the local store. If no suchobject A0 exists then send A to the remote store and record it locallyas having been sent. To send a local object to the remote store, atypical method as embodied here is: send all the hashes and offsets ofdata chunks within the object; query the remote store as to which hashesrepresent data that is not present remotely; send the required data tothe remote store (sending the data and hashes is implemented in thisembodiment by encapsulating them in a TCP data stream).

Conversely, if A0 is identified, then run the difference engine toidentify data chunks that are in A but not in A0. This should be asuperset of the data that needs to be sent to the remote store. Sendhashes and offsets for chunks that are in A but not in A0. Query theremote store as to which hashes represent data that is not presentremotely; send the required data to the remote store.

Sample Deployment Architecture

FIG. 16 shows the software and hardware components in one embodiment ofthe Data Management Virtualization (DMV) system. The software in thesystem executes as three distributed components:

The Host Agent software 1602 a, 1602 b, 1602 c implements some of theapplication-specific module described above. It executes on the sameservers 1610 a, 1610 b, 1610 c as the application whose data is undermanagement.

The DMV server software 1604 a, 1604 b implements the remainder of thesystem as described here. It runs on a set of Linux servers 1612, 1614that also provide highly available virtualized storage services.

The system is controlled by Management Client software 1606 that runs ona desktop or laptop computer 1620.

These software components communicate with one another via networkconnections over an IP network 1628. Data Management Virtualizationsystems communicate with one another between primary site 1622 and datareplication (DR) site 1624 over an IP network such as a public internetbackbone.

The DMV systems at primary and DR sites access one or more SAN storagesystems 1616, 1618 via a fibre-channel network 1626. The servers runningprimary applications access the storage virtualized by the DMV systemsaccess the storage via fibre-channel over the fibre-channel network, oriSCSI over the IP network. The DMV system at the remote DR site runs aparallel instance of DMV server software 1604 c on Linux server 1628.Linux server 1628 may also be an Amazon Web Services EC2 instance orother similar cloud computational resource.

FIG. 17 is a diagram that depicts the various components of acomputerized system upon which certain elements may be implemented,according to certain embodiments of the invention. The logical modulesdescribed may be implemented on a host computer 1701 that containsvolatile memory 1702, a persistent storage device such as a hard drive,1708, a processor, 1703, and a network interface, 1704. Using thenetwork interface, the system computer can interact with storage pools1705, 1706 over a SAN or Fibre Channel device, among other embodiments.Although FIG. 17 illustrates a system in which the system computer isseparate from the various storage pools, some or all of the storagepools may be housed within the host computer, eliminating the need for anetwork interface. The programmatic processes may be executed on asingle host, as shown in FIG. 17, or they may be distributed acrossmultiple hosts.

The host computer shown in FIG. 17 may serve as an administrativeworkstation, or may implement the application and Application SpecificAgent 402, or may implement any and all logical modules described inthis specification, including the Data Virtualization System itself, ormay serve as a storage controller for exposing storage pools of physicalmedia to the system. Workstations may be connected to a graphicaldisplay device, 1707, and to input devices such as a mouse 1709 and akeyboard 1710. Alternately, the active user's workstation may include ahandheld device.

Throughout this specification we refer to software components, but allreferences to software components are intended to apply to softwarerunning on hardware. Likewise, objects and data structures referred toin the specification are intended to apply to data structures actuallystored in memory, either volatile or non-volatile. Likewise, servers areintended to apply to software, and engines are intended to apply tosoftware, all running on hardware such as the computer systems describedin FIG. 17.

Data Fingerprint for Copy Accuracy Assurance

FIG. 18 illustrates a method for generating a data fingerprint for anobject stored in a virtual storage pool, according to certainembodiments of the invention.

A data fingerprint is a short binary digest of a data object that may begenerated independently regardless of how the data object is stored, andis identical when generated multiple times against identical input datawith identical parameters. Useful properties for the fingerprint arethat it be of fixed size, that it be fast to generate for data objectsin all storage pools, and that it be unlikely that different dataobjects have identical fingerprints.

A data fingerprint is different from a checksum or a hash. For example,a fingerprint is taken for only a sample of the object, not the wholeobject. Obtaining a binary digest of a small percentage of the dataobject is sufficient to provide a fingerprint for the whole data object.Since a data fingerprint only requires reads and computes on a smallpercentage of data, such fingerprints are computationally cheap orefficient compared to a checksum or hash.

These data fingerprints are also different in that a single data objectmay have multiple fingerprints. Over the life of a data object, multiplefingerprints are stored with the object as metadata, one per generationof the data object. The multiple fingerprints persist over multiplecopies and generations of the data object.

Data fingerprints may be used to compare two objects to determinewhether they are the same data object. If the data fingerprints for twoobjects differ, the two objects can definitively be said to bedifferent. As with checksums, data fingerprints may thus be used toprovide a measure or test of data integrity between copied or storedversions of a data object. Two data objects with the same datafingerprint may not necessarily be the same object.

As multiple fingerprints are taken of an object, data fingerprints maybe used to compare two objects with increasing reliability. Afingerprint match on a subsequent revision increases confidence that allthe previous copies were accurate. If a fingerprint does not match, thisindicates that either this copy or previous copies were not accurate.With each next generation of the copy, a new fingerprint may be computedand validated against the corresponding fingerprint for that generationor revision.

If two data objects are compared by comparing their corresponding datafingerprints, and the corresponding fingerprints do not match, it ispossible to conclude with certainty that the two data objects aredifferent. However, if the corresponding fingerprints do match, it isnot possible to conclude that the corresponding data objects arenecessarily identical. For example, given two data objects thatrepresent a digital photograph or image data, taking a data fingerprintmay include taking a checksum or binary digest of a portion of eachimage. Comparing the two data objects based on a single portion of eachimage would not necessarily indicate that they are the same image.However, if multiple portions of the two images are identical, it ispossible to conclude with increased certainty that the two images arethe same image.

The calculation of a data fingerprint may require a selection function,which may be dynamic, that selects a subset or portion of the input dataobject. Any such function may be used; one specific example is describedbelow in connection with certain embodiments. The function may selectsmall portions of the data object that are spread out throughout theentirety of the data object. This strategy for selecting portions ofdata is useful for typical storage workloads, in which large chunks ofdata are often modified at one time; by selecting a relatively largenumber of non-contiguous portions or extents of data that are widelydistributed within the data object, the selection function increases theprobability that a large contiguous change in the data object may bedetected. The function may change over time or may base its output onvarious inputs or parameters.

The choice of a selection function should ideally be done with anawareness s of the content of a data object. Portions of the data objectthat are likely to change from generation to generation should beincluded in the fingerprint computation. Portions of the data objectthat are static, or tend to be identical for similar objects should notbe included in the fingerprint. For example, disk labels and partitiontables, which tend to be static should not generally be included in thefingerprint, since these would match across many generations of the sameobject. The tail end of a volume containing filesystems often tend to beunused space; this area should not be used in the computation of thefingerprint, as it will add computational and IO cost to thefingerprint, without increasing its discriminating value.

It is apparent that as the total size of the subset selected by theselection function increases, the probability that the data fingerprintcaptures all changes to the data increases, until the subset is equal tothe whole data object, at which time the probability is 1. However, theselection function may balance the goal of increased probability ofdetecting changes with the goal of providing a consistently-fastfingerprinting time. This tradeoff is expressly permitted, as thedisclosed system allows for multiple data fingerprints to be taken ofthe same data object. Multiple fingerprints can provide the increasederror-checking probability as well, as when the number of fingerprintsbecomes large, the number of un-checked bytes in the data objectdecreases to zero.

A data fingerprinting function may operate as follows, in someembodiments. A data object, 1810, is any file stored within any virtualstorage pool, for example a disk image stored as part of a dataprotection or archiving workflow. Start, 1820, is a number representingan offset or location within the file. Period, 1830, is a numberrepresenting a distance between offsets within the file. Data Sample,1840, is a subset of data from within the data object. Chunk checksums,1850, are the result of specific arithmetic checksum operations appliedto specific data within the file. The data fingerprint, 1860, is asingle numerical value derived deterministically from the content of thedata object 1810 and the parameters start 1820 and period 1830. Otherparameters and other parametrized functions may be used in certainembodiments.

The data samples 1840 are broken into fixed length chunks, in thisillustration 4 KB. For each chunk a chunk checksum 1850 is calculatedfor the data stream, where the checksum includes the data in the chunkand the SHA-1 hash of the data in the chunk. One checksum algorithm usedis the fletcher-32 method(http://en.wikipedia.org/wiki/Fletcher's_checksum). These chunkchecksums are then added together modulo 2⁶⁴, and the arithmetic sum ofthe chunk checksums is the data fingerprint 1860, parameterized by Startand Period. Other methods for combining the plurality of hash values orchecksums into a single hash value may be contemplated in certainembodiments of the invention. A single hash value is preferred forsimplicity. It is not necessary for the single hash value to revealwhich data subsets were used in producing the chunk checksums.

In other embodiments, a data fingerprint may be performed using otherfunctions that focus on interesting sections of a data object, wherecertain sections are determined to be interesting using various means.Interesting sections may be sections that are determined to changefrequently, or that are likely to change frequently. A prioriinformation about the content of the data object or the frequency ofchange of parts of the data object may be used. For example, when thesystem detects that a data object is a disk image, the system may ignorethe volume partition map, as the partition map rarely changes. Asanother exanple, if the system knows that it is storing a Microsoft Worddocument, and that the headers of the document are unlikely to change,it may designate the body and text areas of the document as“interesting,” and may choose to fingerprint those areas. Fingerprintingan “interesting” area may be performed in a manner similar to FIG. 18,in some embodiments, where the data samples are chosen by firstidentifying interesting data areas and then identifying areas to samplewithin the interesting data areas using an algorithm that generates asparse subset of the interesting data areas.

In a preferred embodiment, the described fingerprinting algorithm has avery small overhead, and thus fingerprinting may be performed often.However, in cases such as when a pool includes offline tapes,fingerprinting all data may not have a reasonable overhead.

FIG. 19 illustrates how the data fingerprint is used for assurance ofaccuracy in copy operations, according to certain embodiments of theinvention.

In addition to the operations described above for the Object Manager501, an additional operation is defined: that of generating afingerprint for a data object, given a set of parameters (operation1930). Every data object that is cataloged is fingerprinted and thefingerprint is stored with all other metadata.

When an object is cataloged, Object Manager 501 may make a request for afingerprint on a data object to each pool. The first fingerprint isgenerated at the first storage-optimized pool or snapshot pool andstored in the catalog store. After a data object is first copied intothe Performance Optimized Pool 508 using the lightweight snapshotoperation, the data movement requestor 912 generates a set of parametersfor a fingerprint, and uses them to request a fingerprint (operation1910) from the object manager 501. In turn the object manager requests afingerprint from the performance optimized pool (operation 1940). Theperformance optimized pool is capable of generating the fingerprint. Ina preferred embodiment, every pool managed by pool manager 504 iscapable of generating a fingerprint. The new fingerprint is stored intothe protection catalog store 908, along with other metadata for theobject as described above (operation 1930).

After any subsequent copy request (operation 1910), such as copy tocapacity optimized pool (operation 1950), the fingerprint is requestedfrom the target pool for the target object (operation 1930, operation1960). Once generated, the stored fingerprint is then passed on to eachsubsequent pool, where the newly calculated fingerprint is then verifiedagainst the stored fingerprint to assure that copying errors have notoccurred. Each subsequent pool may calculate the fingerprint again andvalidate the calculated fingerprint against the stored fingerprint.

To generate a fingerprint, the data object 1810 is sampled at regularintervals defined by Start 1820 and Period 1830 parameters. Each sampleis a fixed size, in this illustration 64 KB. In one embodiment, theparameter Period is chosen such that it is approximately 1/1000 of thesize of the data object, and Start is chosen between 0 and Periodaccording to a pseudo-random number generator.

For each new revision or generation of the data object, the startparameter may be modified, resulting in a data fingerprint of adifferent region of the data object. The object size, however, changesonly in certain circumstances. If the object size stays constant theperiod stays constant. If the object size changes the period will changeas well. A period of 1/1000 (0.001) or another small fraction may beselected to ensure that calculating a fingerprint will take a small timeand/or a constant time. Note that depending on the function used togenerate the subset of the data object used for the data fingerprintingoperation, other parameters may be modified instead of the startparameter. The result is to cause the data fingerprint to be generatedfrom a different region of the data object, such that cumulative datafingerprints result in fingerprinting of an increasing proportion of thedata object over time.

Multiple generations of a data object may be created as a result ofinteractions with service level agreements (SLAs), as describedelsewhere in the present disclosure. For example, given a SLA thatschedules a snapshot operation once every hour, an additional generationof a data object will be created every hour. For each additionalgeneration, a new data fingerprint is created and sent. If the dataobject has not changed from the previous generation to the currentgeneration, the data itself need not be sent, but a fingerprint is sentto the target data pool regardless, to incrementally increase theprobability that the sparse data fingerprinting operation has capturedall changes to the data throughout the data object.

As different storage pools may support different operations, thefingerprint operation may be supported by one or more storage pools inthe system. The pools are brokered by the operation manager such as PoolRequest Broker 602. In a preferred embodiment the fingerprint operationis supported by all pools.

Fingerprinting remains with the metadata for the lifetime of the dataobject. This allows fingerprinting to also be used during restore aswell as during copy or other phases of data storage, access andrecovery, which provides true end-to-end metadata from a dataperspective. Fingerprinting during restore is performed as follows. Whena restore operation is requested by Object Manager 501, a fingerprintoperation may take place on the restored data. This fingerprintoperation may take place before or after the restore operation. By usingthe fingerprint operation, all previously-stored revisions of the dataobject are used to verify the currently-restored copy of the data,according to the fingerprint verification method described above. Thisleverages incremental knowledge in a way different from that of typicalI/O path CRC protection.

As disclosed above, each copy of an object between virtual storage poolsis incremental, transferring only data from the source object known tobe absent in the target pool. It follows from this that any errors incopying in one generation of an object will still be present insubsequent generations. Indeed such errors may be compounded. The use ofa data fingerprint provides a check that copies of an object indifferent virtual storage pools have the same data content.

The choice of data fingerprint method also controls the level ofconfidence in the check: as the Period (1830) is made smaller, the costof generating the fingerprint goes up, as more data needs to be readfrom the pool, but the chance of generating a matching fingerprintdespite the data containing copying errors decreases.

However, for successive generations of a single object, different valuesmay be used for the parameter Start (1820). This ensures that withrepeated copying of successive generations of single object, the chancethat any copying error might not be eventually caught reducesasymptotically to zero.

Hybrid Seeding for Improved Incremental Copy Performance

When data management virtualization is used, it becomes possible toseparate different operations required during of a data object'slifecycle between pools, thereby utilizing the best qualities of eachpool to provide superior performance. Just as objects are stored inmultiple places because each pool has different characteristics, otheroperations may also be based on the best characteristics of each placethe data is stored. In the method described below, if it is faster totransfer data out of a given pool, that pool can be used to transfer thedata. This is effectively performing a type of multiplexing based onattributes of the pool. Different pools may be selected based on thepool that is best for the specific situation.

FIG. 20 illustrates an improved method for data backup and restore usingthe Object Manager and Data Mover in some embodiments, where copies ofthe object to be moved already exist in more than one virtual storagepool.

The procedure described below is called hybrid seeding, and is namedaccording to the fact that the data is copied, or “seeded,” to aplurality of storage pools, which are later used to copy the data toanother pool. The greatest value is when there is a large amount of datato be transferred. However, the cost for performing this method is low,so this method may be used for both large and small transfers.

Performance Optimized Pool 2010 is a virtual storage pool with theproperty that retrieving data and metadata from an object in the pool isquick, for example a snapshot pool.

Capacity Optimized Pool 2020 is a virtual storage pool with slower data-and meta-data retrieval characteristics, for example a contentaddressable store. Target Pool 2030 is a virtual storage pool that is atarget for copying data from pool 2020, for example, a contentaddressable store on a remote system. Each pool has differentperformance characteristics: the capacity optimized pool performs wellat performing differences and the performance optimized pool performswell at retrieving bulk data. If one pool performs better than anotherpool at a particular task, it may be said to have relatively highperformance at that task relative to the other pool.

Object A2, 2040, is a data object that has previously been copied intostorage pool 2020. Object A3, 2050, is the result of a previous copy ofobject A2 into storage pool 2030. Object B1 2060, is a data object instorage pool 2010 that is the result of changes to an object A, that is,it is a newer generation or version of A. Object B2, 2070, is the resultof a previous copy of object B1 into storage pool 2020. Object B3, 2080,is the intended result of a copy of object B2 into target pool 2030.

Object Manager 501 (not shown) acts as a controller to direct a commandto copy data objects to the data stores which contain the data objects.In this case, the copy operation for Object B2, 2070, being copied fromcapacity-optimized pool 2020 to capacity-optimized pool 2030, isoptimized using hybrid seeding.

In operation 2100, the Difference Engine 614 is instructed by the ObjectManager to compute differences between objects A2 and B2 prior tocopying the differences to target pool 2030 to be applied to an objectthere. This results in the execution of logic that determines thatstorage pool 2020 is capable of higher-performance differencing thanstorage pool 2030 and therefore that storage pool 2020 should performthe differencing operation. This logic may reside in storage pool 2020,or may also reside in Object Manager 501, or may reside elsewhere in thestorage virtualization system. Since A2 has previously been copied tostorage pool 2030 as A3, and since B2 has previously been copied fromstorage pool 2010 to pool 2020 as B1, the result of operation 2100 isalso a delta of object A3 and B1. This delta or difference set may becharacterized as a type of differences specification.

In operation 2200, the delta of A3 and B1 (not shown) is requested bythe Object Manager to be copied from storage pool 2010 to storage pool2030, to be applied to object A3 in accordance with differences computedpreviously at storage pool 2020. The copy may be performed via directconnection between storage pool 2010 and storage pool 2030, in someembodiments. This division of the differencing operation and the bulkdata copy operation results in a higher-performance differencingoperation at capacity-optimized pool 2020 than would have been possibleat performance-optimized pool 2010. The division also results in ahigher-performance copy from performance-optimized pool 2010 than wouldhave been possible had the copy been performed from capacity-optimizedpool 2020.

In certain embodiments, the logic for each storage pool may be providedby a single centralized controller, in which case the messagingdescribed here may occur within the controller; in other embodiments,logic for one or more storage modules may be executed on computingresources at the storage pools themselves. Copy operations may berequested by the Data Mover or by capacity-optimized pool 2020 or otherpools in some embodiments.

This completes the copy of all data that has changed since A from pool2010 to pool 2030, and results in objects B1, B2, and B3 beingsynchronized between pools 2010, 2020 and 2030.

This method is applicable at least under the following preconditions:Object A has identical copies in two pools, copy A2 (2040) in a capacityoptimized pool 2020 and copy A3 (2050) in another target capacityoptimized pool 2030. These may for example be the result of previouslycopying an object A from storage pool 2010 via pool 2020 to pool 2030.Object B is a newer version of A, and has identical copies in two pools,copy B1 in the first performance optimized pool 2010 and copy B2 in thecapacity optimized pool 2020. For example, B has been copied alreadyfrom pool 2010 to pool 2020 as a backup. And Object B is to be copiedfrom the first capacity optimized pool 2020 to the second, the targetpool 2030. The retrieval time for data from the first pool 2010 is muchbetter than that for the second pool 2020. For example, pool 2010 isbased on enterprise-class primary storage, while pool 2020 is a lowercost or higher latency device more suited to archiving or backup.

This method improves the overall object copy time from the second poolto the third pool. As described previously the copy is executed byinvoking the differencing engine in pool 2020 to provide a set ofdifferences between A2 and B2. Specifically, the differences generatedby the difference engine are a description of the changed sections ofthe object, not the data themselves. For example, the differences mayinclude a set of (offset, length) pairs describing extents within thedata object that have changed.

In prior implementations the differences between sections would begenerated by reading from object B2 and object B3, and then applying thedifferences to object B3 to achieve the required copy. However, in themethod now illustrated, the differences are generated by the differenceengine in pool 2020, but the data are read from the object in pool 2010,that is, the sections are read from object B1 and B2 and then applied toobject B3 to achieve the required copy.

Since one precondition was that retrieval time from pool 2010 be betterthan from pool 2020, this affords improved copy rate.

A similar method may also be used during restore operations. Whenperforming a restore, differencing and bulk data copy may be separated,and bulk copying may be performed from the data store that is fastest,rather than from the store that performs the differencing operation.

In another general sense, this separation of operations by poolconstitutes a hybridization of operations. Different operations areperformed at different storage pools, and these operations are combinedinto a single operation by the virtualization layer. Further suchapplications may exist. For example, comparing two objects in theperformance pool may be a difficult task, but the comparison operationsmay be performed at two content-addressable pools containing the samedata objects.

Replication for Business Continuity

FIG. 21 illustrates a mechanism for data replication for DisasterRecovery and Business Continuity according to some embodiments.

Replication and failover are well-understood operations in whichbusiness logic and data are maintained at a hot backup at a remote site.Failover transfers operation over to the remote site. Replicationtransfers data over to the remote site. A new replication method isdescribed below by which a pipeline of storage pools is used inconjunction with data management virtualization to reduce the amount ofdata that gets transferred. This method also enables bidirectionalcontinuous deduplicated replication.

Business continuity and disaster recovery are well established practicesin the IT industry. For operation they depend on having data from aprimary location replicated to a secondary location regularly. Thereplica at the secondary location must be consistent, that is, it mustrepresent a state of the data as it was at some moment in time at theprimary location such that it can be used to start a secondaryapplication server at the secondary location. The replica needs to bequick to access, so that a secondary application server can be startedup using the data very quickly in the event of the primary applicationserver becoming unavailable. And the replica needs to be low-latency,i.e., when data is made available to an application server at thesecondary location it should represent a consistency point at a time onthe primary server that is as recent as possible, typically measured inminutes.

In addition to replication, a sync-back operation may also be supported.Sync-back is the operation that is performed when data is transferredback from the backup site to the main site. In other words, thedirection of the data copy arrow points in the opposite direction.Sync-back supports and enables fail-back, which is an operation thatreverses the fail-over and transfers operation back to the primaryserver.

It is necessary to be able to perform multiple sync-backs becausetypically business applications keep running while the sync is occuring.It is therefore necessary for the sync-back to be efficient.

Primary Location 2100 is a location where a business suppliesapplication or data services.

Primary Application Server 2101 is a server at location 2100,representative of one or more such servers, delivering business servicesthat may be consumed locally or remotely over some network interface.

Standby Location 2110 is a location where the business may alternativelysupply application or data services, in some event that causesdisruption to the availability of service at the primary location. Forexample, a power outage at the primary location might cause a web serverto be unavailable, in which case a web server at the backup locationcould be configured to respond to the requests that would normally bedirected to the primary location.

Primary Data A, 2120, is the live data being read and written for theoperation of the primary application server. This might for exampleinclude a database that is servicing transactions for a web applicationinterface. In a preferred embodiment, this is a LUN exported to a theprimary application server over a storage network. In another preferredembodiment, this is a disk image for a virtual machine.

At primary location 2100, Primary Pool 2130 is the primary storageresource pool from which storage for the operation of the primaryapplication server is allocated. This would typically be an enterpriseclass SAN storage array. Performance Optimized Pool, 2131, is a storagepool for data protection as described previously which supports thelightweight snapshot operation and differencing. In a preferredimplementation this is a snapshot pool based on low cost networkedstorage. Capacity Optimized Pool, 2132, is a storage pool that supportsthe differencing operation. In a preferred implementation this is adeduplicating content addressable store.

At standby location 2110, Capacity Optimized Pool, 2133 is a storagepool at the standby location, that in turn supports the differencingoperation. Again, in a preferred implementation this is a deduplicatingcontent addressable store. Performance Optimized Pool, 2134, is astorage pool at the standby location that has faster access times thanis typically the case for a content addressable store. In a preferredimplementation, this is a snapshot pool based on low cost networkedstorage. Primary Pool 2135 is a storage pool from which storage can beallocated for execution of a standby application server. This could, forexample, be an enterprise class SAN storage array.

Copies A1 2121, A2 2122, A3 2123, A4 2124, A5 2125 of the primary dataobject A are exact data copies to be created within each of the storagepools as described. The sequential copy operations 2140 will bedescribed below in greater detail. These operations are issued by theService Policy Engine 501 and brokered to each storage pool by theObject Manager 501 as described previously. Logic implementing theService Policy Engine, Object Manager, and other controllers may beimplemented in a centralized server or may be distributed across thenetwork.

The purpose of this method of combining operations and components asdescribed previously is to meet these goals for business continuity,while reducing the load on a network connection, and not requiring anydedicated network bandwidth for business continuity in addition toreplication for data protection. Stated differently, the purpose may beto provide the effect of asynchronous mirroring of data from a primarylocation to a second location.

There are three operational flows to be outlined:

Regular data replication

Failover: using replicated data

Sync-back: posting standby changes back to primary pool

1. Regular Data Replication

The Service Policy Engine is responsible for marshalling operations insequence such that virtual copies of Primary Data A are created in eachpool in the sequence. As the Service Policy Engine issues copy requeststo the Object Manager, the Object Manager brokers these requests tolightweight snapshot or efficient incremental copy operations betweenpairs of pools. Thus, the first operation executed is to make alightweight snapshot of the current state of Primary Data A; the secondoperation executed is to copy just the changed extents within SnapshotA1 into the Capacity Optimized Pool, generating a new content-addressedobject A2; the third operation is to use efficient replication betweencontent addressable stores to generate a new content addressed object A3with minimized data transfer due to data deduplication in the firstcapacity-optimized pool; the fourth operation is to apply just thechanges between A3 and its previous revision in order to update aprevious object in the second Performance Optimized Pool to A4.

At the time the virtual A4 has been completely defined, and in apreferred implementation, verified using a fingerprint-matchingmechanism as described previously, some previous revisions of the sameobject may now be removed from the second Performance Optimized Pool andfrom other intermediary pools. In the preferred implementation, thelatest revision and one revision back are retained, while olderrevisions are removed. This allows the differencing engineimplementations in each pool to find adequately close matches each timea new virtual copy is to be transferred.

In some embodiments, capacity-optimized pool 2133 at the standbylocation may receive metadata from capacity-optimized pool 2132 at theprimary location, and data from performance-optimized pool 2131 at theprimary location, thus providing faster throughput for the bulk datatransfer. In other embodiments, capacity-optimized pool 2133 may receivedata from the primary location and may immediately send it toperformance-optimized pool 2134. In other embodiments, one or both ofcapacity-optimized pool 2132 or performance-optimized pool 2131 at theprimary location may send the metadata and data to the remote location,with both capacity-optimized pool 2133 and performance-optimized pool2134 as the destination. Designating performance-optimized pool 2134 asthe destination and immediately copying the data toperformance-optimized pool 2134 provides updated data to the remoteperformance pool in as short a time as possible, reducing the timewindow between backups during which data loss can occur, and allowingthe remote failover location to resume operation with as little lostdata as possible.

Since the remote performance-optimized pool always has at least onecomplete older copy of the data on the original system, access to dataon the remote pool can be provided near-instantaneously. The data on theremote site does not depend on the data at the local site, and is storedin a native format that is readily usable by the business application.In the event of a failover, the data on the remote performance-optimizedpool is available within a finite length of time, and the length of timeis independent of the size of data stored and of the latency oravailability of the data link between the local site and the remotesite. Since data is stored in a native format at the remote site, it ispossible to copy data directly in a native format between the local siteand the remote site, in some embodiments.

In a preferred implementation, these virtual copy operations arescheduled to execute successively or serially with a delay or with nodelay between them; the entire sequence to be initiated at a regularinterval which may be selected by an operator to vary from minutes tohours.

In another implementation the entire sequence is programmed torecommence as soon as it completes.

In another implementation the operations are pipelined tightly sooperations near the start of the sequence overlap operations near theend of the sequence. This reduces latency while incurring the greatestresource consumption.

In other implementations, the system operates in a parallel fashion, sothat multiple operations may occur simultaneously.

2. Failover—Using Replicated Data

In the event that the Primary Application Server becomes damaged orotherwise unavailable, the standby application server must be broughtinto operation with known-good data from the primary side that is asrecent as possible. In the mechanism described here, a virtual copy ismade of the most recent data object in the standby Performance OptimizedPool, to obtain a data object which can once again be used as a diskimage or logical unit number (LUN) that may be referenced and modifiedby the standby application server. In the preferred implementation, thevirtual copy is made into a standby primary pool using the lightweightcopy operation.

3. Sync-back: Posting Standby Changes Back To Primary Pool

Typically, the standby site is not as well provisioned with resources orconnectivity as the primary site, so once the primary site is availableonce more it is preferred that the business service in question be onceagain provided at the primary site. However any changes made to datawhile the secondary site was providing the business service mustthemselves be replicated back to the primary site.

In this mechanism, this is achieved by a sequence of efficient virtualcopy operations from the modified version of A5 thru to a new version ofprimary data A thus:

B4, a space efficient snapshot of B5, the modified version of A5, ismade in the standby performance optimized pool 2134.

B3, a content-addressed object in the standby capacity optimized pool ismade by efficient copy of changed extents only in B4.

B2, a copy of B3, is made in the primary capacity optimized pool byminimal transfer of deduplicated data.

Bi in the primary performance optimized pool is generated efficiently byefficient copy using the difference engine.

Bi is then available for an operator to restore to the primaryapplication server in a variety of ways. In a preferred implementation,a virtual copy of B̂ is made using a lightweight snapshot operation andexposed as a LUN or disk image to the primary application server. Inanother implementation the data from B̂ is fully copied over LUN or diskimage A. In another implementation, a virtual copy of BA is made using alightweight snapshot operation and exposed as a LUN or disk image to anew primary application server which may be a physical server or virtualmachine.

This method effectively allows the failover/replication site primarypool 2135 to be a high-performance data store providing instant accessor very short access time and efficient and recent transfer of primarypool 2130 data, while still providing the benefits of the datamanagement virtualization system. The optimization is enabled by the useof the intelligent deduplication and other methods described above.

Orchestrating Failovers and Test Failovers

Products and features that leverage the portability of VMs to helpfacilitate the disaster recovery process typically fall into one of twocategories. First are the products that are primarily software-driven.These products facilitate replication of virtual machines, at a per-VMlevel, from a production site to a Disaster Recovery (DR) site.Typically these products meet the need of customers with a small numberof data and virtual machines, but do not scale to handle hundreds of VMsand multiple TB of data. Second are the products that rely upon storagearray replication to get data from the primary site to the DR site.These products also orchestrate the recovery of VMs in a pre-determinedorder, and some have the ability to reconfigure the VMs like thesoftware products can. However, these products often have a requirementto perform fail-over at a granularity decided by the storagearchitecture, as all VMs on the same storage volumes need to be failedover together. While this approach providers greater scalability thanthe software products, it lacks in the flexibility needed to supportcustomer needs without driving up administrative burden and loweringstorage efficiency.

In some embodiments of the present disclosure, systems and methods caninclude a software product (also referred to herein as ResiliencyDirector or “RD”) that works in conjunction with a data managementsystem (e.g., a Dedup Async replication product) to provide orchestratedDisaster Recovery (DR) (also referred to herein in some embodiments as“recovery,” “failover” or “test failover”). The systems and methodsdescribed herein can overcome the limitations of existing DRorchestration tools by providing the flexibility of a software-drivenapproach combined with the scalability of storage array replication.

In some embodiments, RD goes beyond other orchestration tools inapproach by allowing the user more control over the order in whichvirtual machines (VMs) are recovered, as well as control over which VMsmay be recovered in parallel vs. sequentially. In some embodiments, VMsare reconfigured during the recovery process to modify settings that aredifferent in the DR site, such as Internet Protocol (IP) address andDomain Name System (DNS) server. Additionally, in some embodiments, RDcan orchestrate the execution of user provided scripts both inside theVMs being recovered and on one or more separate servers at each stage ofthe recovery process.

FIG. 22 is a system diagram showing the components involved in an RDdeployment, according to some embodiments of the present disclosure.FIG. 22 shows a source location 2201 and a DR site 2202. Source site2201 includes a hypervisor infrastructure 2211, a data managementappliance 2212, and RD Collector 2213. DR site includes a Hypervisorinfrastructure for recovery 2214, a data management appliance 2215, andRD Server 2216.

At the source site 2201 there is an infrastructure running virtualmachines (VMs) in a Hypervisor 2211. This is where the productionservers and data reside. These VMs can be recovered at the DR site 2202,with recovery orchestrated by RD, and facilitated by both datamanagement appliance (e.g., Actifio CDS, Actifio Sky) and RD.

Data Management appliance (also referred to herein as, but not limitedto Actifio CDS or Sky appliance) 2212 at the source site captures theproduction VMs 2220. In some embodiments, data is replicated withActifio Dedup Async Replication (DAR) 2227.

CDS or Sky appliance 2215 at DR site 2202 receives the replicated data.This data is stored in a usable format so that it is available forimmediate use at the DR site. Examples of a usable format are explainedin more detail above. Briefly, usable format include any format where itis not necessary to decompress, rehydrate, or otherwise transform thedata before the VM may be powered on and used with production-likeperformance.

The DR site 2202 houses another infrastructure running a Hypervisor 2214that can be used to run the VMs after a failover or test failover event.This environment is typically sized appropriately to hold all of the VMsin scope for recovery.

The source location has the RD Collector 2213. The RD collector monitorsthe VMs in the Hypervisor 2221 and the configuration of the CDS or Skyappliance 2222. The RD collector allows the user to select VMs that canbe in-scope for recovery, based on the list of VMs in the Hypervisorinfrastructure and the list of VMs being replicated by CDS or Sky usingDedup Async Replication. The user may also configure several aspects ofthe recovery for in-scope selected VMs.

The RD server 2216 resides at the DR site 2202. RD server 2216 pullsconfiguration data from the RD collector 2226. RD server 2216 uses thisinformation, combined with additional user defined configurationinformation, to facilitate the failover process. A failover can becomprised of several steps. In some embodiments, RD server instructs theCDS or Sky appliance to perform a failover 2225. The CDS or Skyappliance then performs the initial failover operation, bringing therecovered VM online on the Hypervisor infrastructure 2224. Finally, theRD server updates VM guest settingson the recovered VM and executes anyuser-defined scripts in those VMs and on any specified additional hosts2223.

FIG. 23 is a component diagram depicting the RD Collector 2213,according to some embodiments of the present disclosure. RD Collector2213 includes application groups 2301, sequences 2302, and VMs 2303.

The RD collector 2213 is a virtual machine that may be running withinthe same or a different hypervisor environment as the production VMsthat are recovered. In some embodiments, RD collector 2213 can beaccessed via web interface to perform configuration tasks.

Users of the RD collector 2213 define one or more Application Groups2301. An application group is a collection of VMs that are recoveredtogether. In some embodiments, for a VM to be eligible for inclusion inan application group the VM is configured for replication on the CDS orSky appliances using Dedup Async Replication. As described in moredetail above, Dedup Async Replication involves providing a remote copyof data associated with virtual machines, in which remote replication isprovided with reduced bandwidth requirements by copying deduplicateddifferences in data from a local storage site to a remote, disasterrecovery site. A more detailed description of Dedup Async Replication isdisclosed in U.S. patent application Ser. No. 14/491,120, filed Sep. 19,2014, entitled “Improved Data Replication System,” the contents of whichare incorporated herein by reference.

Sequences 2302 are defined as part of an application group. Multiplesequences may exist within a single application group. Each sequence isa collection of one or more VMs 2303. When recoveries are performed atthe DR site, all VMs assigned to the same sequence are recovered inparallel. Once all VMs in a sequence are recovered, the next sequence isprocessed.

Each VM 2303 can be flagged to use the same network configuration thatexists in the primary version, or to receive updated networkconfiguration upon failover. Each VM can also have scripts specified tobe executed at the beginning or end of any recovery action for that VM.In some embodiments, the scripts already reside within the VM. The usercan specify the name and type of the script, and the credentials to usewhen executing it. Each VM may optionally be flagged as critical. A VMflagged as critical can impact the entire recovery plan execution ifrecovery of the VM fails, as explained in more detail below.

FIG. 24 is a component diagram depicting the RD Server 2216. FIG. 24shows the Customers 2401, External Scripts 2402, Remote Hosts 2410,Recovery Plans 2403, and Application Groups 2301. The relationshipbetween these components, including the one-to-many nature of somecomponents is also shown.

The RD Server 2216 is a virtual machine that can be running within thesame or a different hypervisor environment as one that is used torecover VMs at the DR site. In some embodiments, it can be accessed viaweb interface to perform configuration tasks.

Users of the RD collector define one or more Customers 2401. A customeris a logical component used to organize the components of the RD Server.It is used used for both reporting and security purposes. RD Collectors2213 that are known to an RD server are associated with a singleCustomer when they are configured. Similarly, recovery plans areassociated with a single Customer when they are created. This allowsreports to be generated showing only recovery plans associated with aspecified customer, and also allows for individual users to be grantedaccess to a subset of RD Collectors and/or recovery plans, which is arequirement when deploying in a multi-tenancy environment.

Remote Hosts 2410 are physical or virtual servers that may be used torun user provided scripts at various times during a recovery planexecution.

External Scripts 2402 are user-provided scripts that may be run atvarious times during a recovery plan execution. External scripts are runon Remote Hosts, and are associated with a single remote host. Externalscripts are typically used when customer-specific applications requirereconfiguration as a part of a recovery process. For example, if anapplication has configuration files that reference IP addresses, and theIP addresses of the VMs are changed as part of the recovery plan, thenexternal scripts may be used to update the configuration files toreference the new IP addresses.

Recovery Plans 2403 are user-defined groupings of items that define whatactions should be taken when performing a recovery. A recovery planincludes one or more application groups to recover.

FIG. 25 is a component diagram depicting a recovery plan 2403. FIG. 25shows a CDS or Sky 2215, Hypervisor environment 2214, RD Collector 2213,Customer 2401, Resource Pool 2501, Default Port Group 2502, and one ormore application groups 2301.

A recovery plan specifies the Actifio CDS or Sky 2215 that has thereplicated VMs to be recovered. In some embodiments, a recovery plan2403 specifies exactly one CDS or Sky. All VMs in all Application Groupsthat are included in the recovery plan can be stored on the specifiedCDS or Sky, and only one is allowed to keep management of theenvironment simple.

A recovery plan specifies the Hypervisor environment 2214 to be usedduring the recovery process. In some embodiments, a recovery planspecifies exactly one Hypervisor environment 2214. All VMs to berecovered can be recovered into the specified hypervisor environment2214. In some embodiments, only one hypervisor environment is specifiedto keep management of resources consumed during recoveries easy tounderstand and monitor by the end-user.

A recovery plan specifies the Customer 2401 to be associated with therecovery plan. In some embodiments, a recovery plan specifies exactlyone Customer. This customer can be used to enable customer-levelrecovery plan reporting, and to allow security restrictions which enableusers to access recovery plans associated with some customers, but notall customers. In some embodiments, only one Customer is specified toensure accuracy of reporting, and proper enforcement of securitypolicies.

A recovery plan specifies the Actifio RD Collector 2213 that has theApplication Group definitions for any application group that can beincluded in a recovery. In some embodiments, one RD Collector isassociated with the recovery plan. The Collector and the Customerassociated with the recovery plan can also be associated with eachother. The limit of one RD Collector per recovery plan can providesimplicity of administration and monitoring of recovery planconfiguration and execution history.

A recovery plan specifies a Resource Pool 2501 available in theHypervisor environment. A resource pool 2501 includes a group of computeresources and optionally some user-defined limits for CPU and memoryresources to be consumed by VMs assigned to the pool. A resource poolcan be associated with a single recovery plan, and a recovery planspecifies one resource pool. The resource pool 2501 is used both toselect which resources within the hypervisor environment can be used forrecoveries by this plan, and also for capacity planning and assurancethat sufficient resources exist in the target hypervisor environment.VMs that are recovered as part of the recovery plan can be placed intothe specified resource pool 2501 within the hypervisor environment.

A recovery plan specifies a default Port Group 2502 available in theHypervisor environment. The port group 2502 can be used as the defaultnetwork to use when recovering VMs. Individual VMs can have a different,non-default port group specified as part of a recovery plan. In someembodiments, a recovery plan specifies one default port group.

A recovery plan also includes one or more application groups 2301. Thisspecifies which VMs 2303 can be recovered as part of the plan, and theorder in which they can be recovered.

A recovery plan can be configured so that each sequence in anapplication group can execute an external script 2402 at the beginningor end of the recovery process for the sequence. This is called thesequence script 2503. A sequence script can be used to performapplication-specific reconfiguration that may be required before orafter all the VMs in the sequence are recovered.

Individual VMs in a recovery plan can be configured to execute anexternal script 2402 at the beginning or end of the recovery process forthe VM 2305. This is called the VM script 2504. A VM script can be usedto perform application-specific reconfiguration that may be requiredbefore or after each VM is recovered.

Individual VMs in a recovery plan can be configured to receive differentconfiguration settings, including the port group, network IP, networksubnet, network default gateway, and DNS server. This is called the VMguest settings 2505.

In some embodiments, a recovery plan can be scheduled to run unattendedon a periodic basis to allow for automated recovery testing.

FIG. 26 is a flow diagram illustrating a process of setting up RD toperform failovers and test failovers, according to some embodiments ofthe present disclosure. FIG. 26 shows setup of the CDS or Sky DedupAsync replication 2601, installation of the RD Collector and Server2602, specification of CDS or Sky to use in RD software 2603,specification of hypervisor environment to use in RD software 2604,definition of one or more application groups 2605 on the RD collector,definition of one or more customers 2606 on the RD server, specificationof zero or more remote hosts 2607 on the RD server, uploading of zero ormore external scripts 2608 to RD server, specification of one or more RDCollectors 2609 on RD server, and definition of one or more recoveryplans 2610 on the RD server.

Dedup Async replication 2601 can be setup using CDS or Sky appliances.In some embodiments, RD relies on this replication as a data transportmethod to get a usable copy of in-scope VMs from the source location tothe DR site.

RD Installation 2602 is performed by creating VMs from the RDinstallation media and configuring them with network connectivity in thehypervisor infrastructure at both source and DR locations. In someembodiments, an RD installation process is identical for RD Collectorand RD Server, with the only exception being the user specifies duringinstallation if the resulting VM is a Collector or Server.

In some embodiments, both RD Server and RD Collector have one or moreActifio CDS or Sky appliances to be specified 2603. Specifying a datamanagement system is done before application groups are defined on thecollector, or recovery plans defined on the server. In some embodiments,a process of specifying a CDS or Sky appliance is identical on the RDServer and Collector components.

In some embodiments, both RD Server and Collector have one or morehypervisor infrastructures to be specified 2604. Specifying a hypervisorinfrastructure is done before application groups are defined on thecollector, or recovery plans are defined on the server. In someembodiments, a process of specifying a hypervisor environment isidentical on the RD Server and Collector components.

Application Groups 2301 are defined 2605 on the RD Collector. Thesegroups are maintained on the collector by the end-user and change as thesource environment changes. The source environment can change throughthe addition and deletion of virtual machines. In some embodiments, arelationship between VMs can change, which can require movement of VMsbetween sequences. Scripts to run within the VMs being recovered can bedefined as part of the application group, as can credentials to use forexecuting those scripts. In some embodiments, the user can also specifyas part of the application group if they wish to override the networksettings at the DR site during recovery or if the recovered VMs shouldhave the same network configuration as at the source. All configurationdata associated with an application group is shared with the RD Serverto be used during recoveries.

User defines one or more Customers 2606 on the RD Server before they canspecify any RD Collectors, or create a recovery plan. In someembodiments, every RD Collector and every recovery plan is associatedwith a Customer, enabling Customer based reporting and security/accessrestrictions.

In some embodiments, user can optionally specify one or more remotehosts 2607 on the RD Server. If specified, the remote host can beavailable for execution of external scripts, should any be uploaded.

In some embodiments, user can optionally upload external scripts 2608 tothe RD server. When uploaded, the scripts are associated with a specificremote host. These scripts are later optionally associated withcomponents of a recovery plan and can be executed during a failover ortest failover operation.

User specifies one or more RD Collectors 2609 on the RD server. Each RDCollector is associated with one Customer. This association enablesCustomer based reporting and security/access restrictions. Allconfiguration data on the defined collectors is imported to the RDServer so that Application Groups defined on the Collector becomeavailable for inclusion in a recovery plan.

User defines a recovery plan 2610 on the RD server. FIG. 25 depicts therecovery plan and all components that need to be defined, and theirpurpose. Once a recovery plan is defined, the user has the ability toexecute the recovery plan, specifying either “failover” or “testfailover” as the execution type.

FIG. 27 is a flow diagram illustrating a process of executing a recoveryplan of either failover or test failover execution type. FIG. 27 showsthe user initiation of a recovery plan 2701, logic for building a listof sequences to process 2702, VM recovery process 2703, checks forcritical failures 2704, looping through all sequences to ensure recoveryof all in-scope VMs 2705 and 2706, and the update of executioncompletion status 2707.

Users have the ability to initiate execution of a recovery plan 2701 onthe RD Server. In some embodiments, users can only initiate execution ofa recovery plan that is in a “Ready to run” state. Other states include“Success”, “Partial Success”, and “Failed”. When executing a recoveryplan, the user specifies if they wish to perform a “failover” or a “testfailover”.

When recovery plan execution begins, the RD Server builds an in-memoryordered list of sequences to be processed 2702. The list is ordered byApplication Group, in the same order they appear in the recovery plan.Within each application group, the sequences are added to the list innumerical order. Processing begins with the first sequence in the firstapplication group in the recovery plan. This first sequence is labeledthe “current sequence” once the list is built and recovery is ready tobegin.

In some embodiments, for each VM in the current sequence, all VMs arerecovered in parallel 2703. The steps involved for each VM, that takeplace in parallel, begin by initiating a “failover” or “test failover”on the CDS or Sky specified in the recovery plan. The action taken,“failover” or “test failover” matches the selection made by the userwhen they initiated the recovery plan 2701. Once the CDS or Sky hascompleted this step, the RD Server initiates any “pre” scripts withinthe VM that have been specified in the recovery plan, apply networkingchanges, if any, and then execute any “post” scripts that have beenspecified in the recovery plan. Additionally, any external scriptsspecified as part of the recovery plan are executed at the timespecified, which can be at the start of processing for the sequence, atthe start of processing for any individual VM, at the end of processingfor any individual VM, or at the end of processing for the sequence.

In some embodiments, the RD server tracks success/failure of each VM'srecovery process. When recovery is complete for all VMs in the sequence,the RD server can check if any VMs with a failed status are flagged as“critical” in the application group 2704. If any critical VMs havefailed their recovery process, the recovery plan execution is halted andthe recovery plan status is updated to “failed” 2707.

If no critical VMs have failed their recovery process, the RD Serverthen determines if any more sequences remain to be processed 2705. Ifthere are some, the RD server advances to the next sequence in the list2706 and repeat the recovery process for all VMs in that sequence, withthe process flow returning to item 2703. This logic repeats until nomore sequences remain to be recovered.

When all sequences have been recovered, or if a sequence has a VMflagged as critical that has failed to recover successfully, the RDServer updates the status of the recovery plan 2707. If all VMs in allsequences have recovered successfully, the recovery plan status can beupdated to “Success”. If some VMs have recovered successfully, and someVMs have failed, but no failed VMs are flagged as critical, the recoveryplan status can be updated to “Partial Success”. If any VM flagged ascritical has failed to recover successfully, the recovery plan statuscan be updated to “Failed”.

After the recovery plan execution is complete, if a recovery plan has astatus of “Partial Success” or “Failed”, the user has the ability toinitiate a single-VM recovery on the RD Server for any VM that has notbeen successfully recovered.

FIG. 28 is a flow diagram illustrating a process of resetting a recoveryplan for either failover or test failover execution type. FIG. 28 showsthe user initiation of a recovery plan reset 2801, logic for building alist of sequences to process 2802, logic to determine what actions totake 2803, VM reset process 2804, looping through all sequences toensure reset of all in-scope VMs 2805 and 2806, and the update ofexecution completion status 2807.

Users have the ability to initiate a recovery plan reset 2801 on the RDServer. Users may only reset a recovery plan for a recovery plan that isin a “Success”, “Partial Success”, or “Failed” state. Other statesinclude “Ready to Run”. When resetting a recovery plan, the result isdifferent depending on the previous execution type and if it was“failover” or “test failover”.

RD Server determines if the previous execution of the recovery plan wasa “Failover” or a “Test Failover” 2802. If it was a “Test Failover”,then process flow skips to step 2807 to update the recovery plan status.

If the previous recovery plan execution was of type “Test Failover”,then the RD Server builds an in-memory ordered list of sequences to bereset 2803. The list is ordered by Application Group, in the reverseorder they appear in the recovery plan. Within each application group,the sequences are added to the list in descending numerical order.Processing begins with the highest sequence in the last applicationgroup in the recovery plan. This sequence can be labeled the “currentsequence” once the list has been built and the reset process is ready tobegin.

For each VM in the current sequence, all VMs can be reset in parallel2804. Resetting a VM involves initiation of a “Delete Test Failover”action on the underlying CDS or Sky appliance and waiting for it tocomplete.

When the reset is complete for all VMs in the sequence, the RD Servercan then determine if any more sequences remain to be processed 2805. Ifany more sequences remain to be processed, the RD server can advance tothe next sequence in the list 2806 and repeat the reset process for allVMs in that sequence, with the process flow returning to item 2804. Thislogic repeats until no more sequences remain to be reset.

When all sequences have been reset, or if the previous recovery planexecution was of type “failover”, the RD Server updates the recoveryplan status to “Ready to Run” 2807.

The subject matter described herein can be implemented in digitalelectronic circuitry, or in computer software, firmware, or hardware,including the structural means disclosed in this specification andstructural equivalents thereof, or in combinations of them. The subjectmatter described herein can be implemented as one or more computerprogram products, such as one or more computer programs tangiblyembodied in an information carrier (e.g., in a machine readable storagedevice), or embodied in a propagated signal, for execution by, or tocontrol the operation of, data processing apparatus (e.g., aprogrammable processor, a computer, or multiple computers). A computerprogram (also known as a program, software, software application, orcode) can be written in any form of programming language, includingcompiled or interpreted languages, and it can be deployed in any form,including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment. Acomputer program does not necessarily correspond to a file. A programcan be stored in a portion of a file that holds other programs or data,in a single file dedicated to the program in question, or in multiplecoordinated files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to beexecuted on one computer or on multiple computers at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification, includingthe method steps of the subject matter described herein, can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions of the subject matter describedherein by operating on input data and generating output. The processesand logic flows can also be performed by, and apparatus of the subjectmatter described herein can be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processor of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for executing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. Information carrierssuitable for embodying computer program instructions and data includeall forms of nonvolatile memory, including by way of examplesemiconductor memory devices, (e.g., EPROM, EEPROM, and flash memorydevices); magnetic disks, (e.g., internal hard disks or removabledisks); magneto optical disks; and optical disks (e.g., CD and DVDdisks). The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, the subject matter describedherein can be implemented on a computer having a display device, e.g., aCRT (cathode ray tube) or LCD (liquid crystal display) monitor, fordisplaying information to the user and a keyboard and a pointing device,(e.g., a mouse or a trackball), by which the user can provide input tothe computer. Other kinds of devices can be used to provide forinteraction with a user as well. For example, feedback provided to theuser can be any form of sensory feedback, (e.g., visual feedback,auditory feedback, or tactile feedback), and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The subject matter described herein can be implemented in a computingsystem that includes a back end component (e.g., a data server), amiddleware component (e.g., an application server), or a front endcomponent (e.g., a client computer having a graphical user interface ora web browser through which a user can interact with an implementationof the subject matter described herein), or any combination of such backend, middleware, and front end components. The components of the systemcan be interconnected by any form or medium of digital datacommunication, e.g., a communication network. Examples of communicationnetworks include a local area network (“LAN”) and a wide area network(“WAN”), e.g., the Internet.

It is to be understood that the disclosed subject matter is not limitedin its application to the details of construction and to thearrangements of the components set forth in the following description orillustrated in the drawings. The disclosed subject matter is capable ofother embodiments and of being practiced and carried out in variousways. Also, it is to be understood that the phraseology and terminologyemployed herein are for the purpose of description and should not beregarded as limiting.

As such, those skilled in the art will appreciate that the conception,upon which this disclosure is based, may readily be utilized as a basisfor the designing of other structures, methods, and systems for carryingout the several purposes of the disclosed subject matter. It isimportant, therefore, that the claims be regarded as including suchequivalent constructions insofar as they do not depart from the spiritand scope of the disclosed subject matter.

Although the disclosed subject matter has been described and illustratedin the foregoing exemplary embodiments, it is understood that thepresent disclosure has been made only by way of example, and thatnumerous changes in the details of implementation of the disclosedsubject matter may be made without departing from the spirit and scopeof the disclosed subject matter, which is limited only by the claimswhich follow.

What is claimed is:
 1. A method of orchestrating recoveries of virtualmachines protected by a plurality of data management systems from aprimary system to a secondary system, such that performing therecoveries depends on relationships between the virtual machines, themethod comprising: receiving, by a computing device, first dataindicative of a recovery plan associated with a failover of at least onegroup of virtual machines, each of the at least one groups of virtualmachines protected by a data management system associated with the atleast one group of virtual machines, the recovery plan comprising: anapplication group, the application group including second dataindicative of a hierarchical relationship between the virtual machinesin the at least one group of virtual machines, wherein each of thevirtual machines in the at least one group of virtual machines isassociated with an order based on the second data; creating, by thecomputing device, a plurality of sequences in the application group todesignate an order of executing a plurality of recoveries for each ofthe virtual machines in the at least one group of virtual machines,wherein the plurality of sequences are ordered for execution based on arelationship between the virtual machines contained with the pluralityof sequences; executing in parallel, by the computing device, a firstrecovery for each of the virtual machines associated with a firstsequence; and executing in parallel, by the computing device, asubsequent recovery for each of a subsequent set of sequences in orderwhen the application group includes a subsequent sequence, such that anorder of performing the recoveries depends on the relationships betweenthe virtual machines.
 2. The method of claim 1, further comprisingupdating, by the computing device, the recovery plan with results of thefailover, the results of the failover comprising a successful recoveryresult, a partially successful recovery result, or a failed recoveryresult.
 3. The method of claim 2, further comprising: determining, bythe computing device, a status of the failover for each of the virtualmachines; terminating, by the computing device, an execution of theplurality of recoveries when the status is associated with a criticalfailure; and updating, by the computing device, the recovery plan withthe failed recovery result.
 4. The method of claim 1, wherein therecovery plan further comprises at least one of: a hypervisor associatedwith the failover; a collector associated with the failover; a customerassociated with the failover; a data management system associated withthe failover; a resource pool associated with the failover; and a portgroup associated with the failover.
 5. The method of claim 1, whereinthe failover is a test failover.
 6. The method of claim 5, furthercomprising resetting the recovery plan, wherein resetting the recoveryplan comprises: receiving, by the computing device, third dataindicative of a recovery plan reset request; executing, by the computingdevice, a delete operation for virtual machines associated with a lastsequence when an execution of the recovery plan associated with a resetis a test failover; and executing, by the computing device, a deleteoperation for virtual machines associated with each of a prior sequencewhen an execution of the recovery plan associated with a reset is a testfailover when the application group includes a prior sequence.
 7. Themethod of claim 6, further comprising updating, by the computing device,the recovery plan with a ready to run status.
 8. The method of claim 1,wherein each of the at least one data management systems protect theplurality of virtual machines using dedup async replication, the dedupasync replication providing a remote copy of fourth data associated withthe virtual machines, in which remote replication is provided withreduced bandwidth requirements by copying deduplicated differences indata from a local storage site to a disaster recovery site.
 9. Themethod of claim 1, for use in virtual machine by virtual machinerecoveries independent of production source architecture, while usingstorage-based replication methodologies.
 10. A computing system fororchestrating recoveries of virtual machines protected by a plurality ofdata management systems from a primary system to a secondary system,such that performing the recoveries depends on relationships between thevirtual machines, the method comprising: a processor; and memory coupledto the processor and including computer-readable instructions that, whenexecuted by a processor, cause the processor to: receive first dataindicative of a recovery plan associated with a failover of at least onegroup of virtual machines, each of the at least one groups of virtualmachines protected by a data management system associated with the atleast one group of virtual machines, the recovery plan comprising: anapplication group, the application group including second dataindicative of a hierarchical relationship between the virtual machinesin the at least one group of virtual machines, wherein each of thevirtual machines in the at least one group of virtual machines isassociated with an order based on the second data; create a plurality ofsequences in the application group to designate an order of executing aplurality of recoveries for each of the virtual machines in the at leastone group of virtual machines, wherein the plurality of sequences areordered for execution based on a relationship between the virtualmachines contained with the plurality of sequences; execute in parallel,by the computing device, a first recovery for each of the virtualmachines associated with a first sequence; and execute in parallel, bythe computing device, a subsequent recovery for each of a subsequent setof sequences in order when the application group includes a subsequentsequence, such that an order of performing the recoveries depends on therelationships between the virtual machines.
 11. The computing system ofclaim 10, wherein the processor is further caused to update the recoveryplan with results of the failover, the results of the failovercomprising a successful recovery result, a partially successful recoveryresult, or a failed recovery result.
 12. The computing system of claim11, wherein the processor is further caused to: determine a status ofthe failover for each of the virtual machines; terminate an execution ofthe plurality of recoveries when the status is associated with acritical failure; and update the recovery plan with the failed recoveryresult.
 13. The computing system of claim 10, wherein the recovery planfurther comprises at least one of: a hypervisor associated with thefailover; a collector associated with the failover; a customerassociated with the failover; a data management system associated withthe failover; a resource pool associated with the failover; and a portgroup associated with the failover.
 14. The computing system of claim10, wherein the failover is a test failover.
 15. The computing system ofclaim 10, wherein the processor is further caused to reset the recoveryplan, wherein resetting the recovery plan comprises: receiving thirddata indicative of a recovery plan reset request; executing a deleteoperation for virtual machines associated with a last sequence when anexecution of the recovery plan associated with a reset is a testfailover; and executing a delete operation for virtual machinesassociated with each of a prior sequence when an execution of therecovery plan associated with a reset is a test failover when theapplication group includes a prior sequence.
 16. The computing system ofclaim 15, wherein the processor is further caused to update the recoveryplan with a ready to run status.
 17. The computing system of claim 10,wherein each of the at least one data management systems protect theplurality of virtual machines using dedup async replication, the dedupasync replication providing a remote copy of fourth data associated withthe virtual machines, in which remote replication is provided withreduced bandwidth requirements by copying deduplicated differences indata from a local storage site to a disaster recovery site.
 18. Thecomputing system of claim 10, for use in virtual machine by virtualmachine recoveries independent of production source architecture, whileusing storage-based replication methodologies.
 19. A non-transitorycomputer readable medium having executable instructions operable tocause an apparatus to: receive first data indicative of a recovery planassociated with a failover of at least one group of virtual machines,each of the at least one groups of virtual machines protected by a datamanagement system associated with the at least one group of virtualmachines, the recovery plan comprising: an application group, theapplication group including second data indicative of a hierarchicalrelationship between the virtual machines in the at least one group ofvirtual machines, wherein each of the virtual machines in the at leastone group of virtual machines is associated with an order based on thesecond data; create a plurality of sequences in the application group todesignate an order of executing a plurality of recoveries for each ofthe virtual machines in the at least one group of virtual machines,wherein the plurality of sequences are ordered for execution based on arelationship between the virtual machines contained with the pluralityof sequences; execute in parallel, by the computing device, a firstrecovery for each of the virtual machines associated with a firstsequence; and execute in parallel, by the computing device, a subsequentrecovery for each of a subsequent set of sequences in order when theapplication group includes a subsequent sequence, such that an order ofperforming the recoveries depends on the relationships between thevirtual machines.
 20. The non-transitory computer readable medium ofclaim 19, wherein each of the at least one data management systemsprotect the plurality of virtual machines using dedup async replication,the dedup async replication providing a remote copy of third dataassociated with the virtual machines, in which remote replication isprovided with reduced bandwidth requirements by copying deduplicateddifferences in data from a local storage site to a disaster recoverysite.